GitBook: [v3.x] 17 pages modified

removed repeated word

.gitbook/assets/convert.md

@@ -0,0 +1,60 @@
+## Yaml to Json
+To convert output to json, use the --tojson (or -j) flag. This is supported by all commands.
+
+Each matching yaml node will be converted to json and printed out on a separate line.
+
+Given a sample.yaml file of:
+```yaml
+b:
+  c: 2
+```
+then
+```bash
+yq r -j sample.yaml
+```
+
+will output
+```json
+{"b":{"c":2}}
+```
+
+Given a sample.yaml file of:
+```yaml
+bob:
+  c: 2
+bab:
+  c: 5
+```
+then
+```bash
+yq r -j sample.yaml b*
+```
+
+will output
+```json
+{"c":2}
+{"c":5}
+```
+
+## Json to Yaml
+To read in json, just pass in a json file instead of yaml, it will just work :)
+
+e.g given a json file
+
+```json
+{"a":"Easy! as one two three","b":{"c":2,"d":[3,4]}}
+```
+then
+```bash
+yq r sample.json
+```
+will output
+```yaml
+a: Easy! as one two three
+b:
+  c: 2
+  d:
+  - 3
+  - 4
+```
+

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,48 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+version of yq: 
+operating system:
+
+**Input Yaml**
+Concise yaml document(s) (as simple as possible to show the bug)
+data1.yml:
+```yaml
+this: should really work
+```
+
+data2.yml:
+```yaml
+but: it strangely didn't
+```
+
+**Command**
+The command you ran:
+```
+yq merge data1.yml data2.yml
+```
+
+**Actual behavior**
+
+```yaml
+cat: meow
+```
+
+**Expected behavior**
+
+```yaml
+this: should really work
+but: it strangely didn't
+```
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,36 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+If we have data1.yml like:
+
+```yaml
+country: Australia
+```
+
+And we run a command:
+
+```bash
+yq predictWeather data1.yml
+```
+
+it could output
+
+```yaml
+temp: 32
+```
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/workflows/go.yml

@@ -0,0 +1,34 @@
+name: Build
+on: [push]
+jobs:
+
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    steps:
+
+    - name: Set up Go 1.14
+      uses: actions/setup-go@v1
+      with:
+        go-version: 1.14
+      id: go
+
+    - name: Check out code into the Go module directory
+      uses: actions/checkout@v2
+
+    - name: Get dependencies
+      run: |
+        go get -v -t -d ./...
+        if [ -f Gopkg.toml ]; then
+            curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
+            dep ensure
+        fi
+
+    - name: Download deps
+      run: |
+        export PATH=${PATH}:`go env GOPATH`/bin
+        scripts/devtools.sh
+    - name: Build
+      run: |
+        export PATH=${PATH}:`go env GOPATH`/bin
+        make local build

.travis.yml

@@ -1,6 +0,0 @@
-language: go
-go:
-  - 1.13.x
-script:
-  - scripts/devtools.sh
-  - make local build

CODE_OF_CONDUCT.md

@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+ advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+ address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at mikefarah@gmail.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

CONTRIBUTING.md

@@ -0,0 +1,8 @@
+1. Install (golang)[https://golang.org/]
+1. Run  `scripts/devtools.sh` to install the required devtools
+2. Run `make [local] vendor` to install the vendor dependencies
+2. Run `make [local] test` to ensure you can run the existing tests
+3. Write unit tests - (see existing examples). Changes will not be accepted without corresponding unit tests.
+4. Make the code changes.
+5. `make [local] test` to lint code and run tests
+6. Profit! ok no profit, but raise a PR and get kudos :)

Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.13 as builder
+FROM golang:1.14 as builder
 
 WORKDIR /go/src/mikefarah/yq
 

Dockerfile.dev

@@ -1,4 +1,4 @@
-FROM golang:1.13
+FROM golang:1.14
 
 COPY scripts/devtools.sh /opt/devtools.sh
 
@@ -9,15 +9,15 @@ RUN set -e -x \
 RUN set -ex \
     && buildDeps=' \
         build-essential \
-        python-dev \
+        python3-dev \
     ' \
     && apt-get update && apt-get install -y --no-install-recommends \
         $buildDeps \
-        python2.7 \
+        python3 \
-        python-setuptools \
+        python3-setuptools \
-        python-wheel \
+        python3-wheel \
-        python-pip \
+        python3-pip \
-    && pip install --upgrade \
+    && pip3 install --upgrade \
         pip \
         'Markdown>=2.6.9' \
         'mkdocs>=0.16.3' \

README.md

@@ -1,56 +1,74 @@
+---
+description: yq is a lightweight and portable command-line YAML processor
+---
+
 # yq
 
-[![Build Status](https://api.travis-ci.com/mikefarah/yq.svg?branch=master)](https://travis-ci.com/mikefarah/yq/)  ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq)
+ ![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg) ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases \(by Release\)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq)
-
-
-a lightweight and portable command-line YAML processor
-
-The aim of the project is to be the [jq](https://github.com/stedolan/jq) or sed of yaml files.
-
-## New version!
-V3 is officially out - if you've been using v2 and want/need to upgrade, checkout the [upgrade guide](https://mikefarah.gitbook.io/yq/upgrading-from-v2).
 
 ## Install
 
+`yq` has pre-built binaries for most platforms - checkout the [releases page](https://github.com/mikefarah/yq/releases) for the latest build. Alternatively - you can use one of the methods below:
+
 ### On MacOS:
-```
+
+```bash
 brew install yq
 ```
-### On Ubuntu and other Linux distros supporting `snap` packages:
+
+### On Windows:
+
+```bash
+choco install yq
 ```
+
+Kindly maintained by @chillum \([https://github.com/chillum/choco-packages/tree/master/yq](https://github.com/chillum/choco-packages/tree/master/yq)\)
+
+### On Ubuntu and other Linux distributions supporting `snap` packages:
+
+```bash
 snap install yq
 ```
 
 #### Snap notes
+
 `yq` installs with with [_strict confinement_](https://docs.snapcraft.io/snap-confinement/6233) in snap, this means it doesn't have direct access to root files. To read root files you can:
 
-```
+```bash
-sudo cat /etc/myfile | yq r - a.path
+sudo cat /etc/myfile | yq -r - somecommand
 ```
 
 And to write to a root file you can either use [sponge](https://linux.die.net/man/1/sponge):
+
+```bash
+sudo cat /etc/myfile | yq -r - somecommand | sudo sponge /etc/myfile
 ```
-sudo cat /etc/myfile | yq w - a.path value | sudo sponge /etc/myfile
+
-```
 or write to a temporary file:
-```
+
-sudo cat /etc/myfile | yq w - a.path value | sudo tee /etc/myfile.tmp
+```bash
+sudo cat /etc/myfile | yq -r - somecommand | sudo tee /etc/myfile.tmp
 sudo mv /etc/myfile.tmp /etc/myfile
 rm /etc/myfile.tmp
 ```
 
 ### On Ubuntu 16.04 or higher from Debian package:
-```
+
+```bash
 sudo add-apt-repository ppa:rmescandon/yq
 sudo apt update
 sudo apt install yq -y
 ```
-### or, [Download latest binary](https://github.com/mikefarah/yq/releases/latest) or alternatively:
+
-```
+Kindly maintained by @rmescandon
+
+### go get:
+
+```text
 GO111MODULE=on go get github.com/mikefarah/yq/v3
 ```
 
-## Run with Docker
+## Docker
 
 Oneshot use:
 
@@ -72,54 +90,3 @@ yq() {
 }
 ```
 
-## Features
-- Written in portable go, so you can download a lovely dependency free binary
-- [Deep read a yaml file with a given path expression](https://mikefarah.gitbook.io/yq/commands/read#basic)
-- [List matching paths of a given path expression](https://mikefarah.gitbook.io/yq/commands/read#path-only)
-- Update a yaml file given a [path expression](https://mikefarah.gitbook.io/yq/commands/write-update#basic) or [script file](https://mikefarah.gitbook.io/yq/commands/write-update#basic)
-- Update creates any missing entries in the path on the fly
-- Keeps yaml formatting and comments when updating
-- Create a yaml file given a [deep path and value](https://mikefarah.gitbook.io/yq/commands/create#creating-a-simple-yaml-file) or a [script file](https://mikefarah.gitbook.io/yq/commands/create#creating-using-a-create-script)
-- [Prefix a path to a yaml file](https://mikefarah.gitbook.io/yq/commands/prefix)
-- [Convert to/from json to yaml](https://mikefarah.gitbook.io/yq/usage/convert)
-- [Pipe data in by using '-'](https://mikefarah.gitbook.io/yq/commands/read#from-stdin)
-- [Merge](https://mikefarah.gitbook.io/yq/commands/merge) multiple yaml files with various options for [overriding](https://mikefarah.gitbook.io/yq/commands/merge#overwrite-values) and [appending](https://mikefarah.gitbook.io/yq/commands/merge#append-values-with-arrays)
-- Supports multiple documents in a single yaml file for [reading](https://mikefarah.gitbook.io/yq/commands/read#multiple-documents), [writing](https://mikefarah.gitbook.io/yq/commands/write-update#multiple-documents) and [merging](https://mikefarah.gitbook.io/yq/commands/merge#multiple-documents)
-
-## [Usage](https://mikefarah.gitbook.io/yq/)
-
-Check out the [documentation](https://mikefarah.gitbook.io/yq/) for more detailed and advanced usage.
-
-```
-Usage:
-  yq [flags]
-  yq [command]
-
-Available Commands:
-  delete      yq d [--inplace/-i] [--doc/-d index] sample.yaml 'b.e(name==fred)'
-  help        Help about any command
-  merge       yq m [--inplace/-i] [--doc/-d index] [--overwrite/-x] [--append/-a] sample.yaml sample2.yaml
-  new         yq n [--script/-s script_file] a.b.c newValue
-  prefix      yq p [--inplace/-i] [--doc/-d index] sample.yaml a.b.c
-  read        yq r [--printMode/-p pv] sample.yaml 'b.e(name==fr*).value'
-  write       yq w [--inplace/-i] [--script/-s script_file] [--doc/-d index] sample.yaml 'b.e(name==fr*).value' newValue
-
-Flags:
-  -h, --help      help for yq
-  -j, --tojson    output as json
-  -v, --verbose   verbose mode
-  -V, --version   Print version information and quit
-
-Use "yq [command] --help" for more information about a command.
-```
-
-## Contribute
-
-**Note: v3 is currently in progress - for the moment I won't be accepting new feature PRs until v3 is ready :)**
-
-1. `scripts/devtools.sh`
-2. `make [local] vendor`
-3. add unit tests
-4. apply changes to go.mod
-5. `make [local] build`
-7. profit

SUMMARY.md

@@ -0,0 +1,25 @@
+# Table of contents
+
+* [yq](README.md)
+* [Upgrading from V2](upgrading-from-v2.md)
+
+## Commands
+
+* [Read](commands/read.md)
+* [Validate](commands/validate.md)
+* [Compare](commands/compare.md)
+* [Write](commands/write-update.md)
+* [Create](commands/create.md)
+* [Delete](commands/delete.md)
+* [Merge](commands/merge.md)
+* [Prefix](commands/prefix.md)
+* [Shell Completion](commands/shell-completion.md)
+
+## Usage
+
+* [Output format](usage/output-format.md)
+* [Path Expressions](usage/path-expressions.md)
+* [Value Parsing](usage/value-parsing.md)
+* [Working with JSON](usage/convert.md)
+* [Github Page](https://github.com/mikefarah/yq)
+

Upgrade Notes

@@ -1,74 +0,0 @@
-
-Major release! Upgraded underlying yaml parser, re-written majority of yq. This has brought on a number of features that have been in demand for a while (see below).
-
-This is in beta and needs some community feedback and testing :)
-
-# New Features
-  - Keeps yaml comments and formatting, can specify yaml tags when updating. https://github.com/mikefarah/yq/issues/19, https://github.com/mikefarah/yq/issues/169, https://github.com/mikefarah/yq/issues/107, https://github.com/mikefarah/yq/issues/171, https://github.com/mikefarah/yq/issues/245, https://github.com/mikefarah/yq/issues/303,https://github.com/mikefarah/yq/issues/308,https://github.com/mikefarah/yq/issues/314
-  - Handles anchors!  https://github.com/mikefarah/yq/issues/310, https://github.com/mikefarah/yq/issues/178
-  - Can print out matching paths and values when splatting https://github.com/mikefarah/yq/issues/20
-  - JSON output works for all commands! Yaml files with multiple documents are printed out as one JSON document per line. 
-  - Deep splat (**) to match arbitrary paths
-
-
-# Breaking changes
-
-## Update scripts file format has changed to be more powerful. 
-Comments can be added, and delete commands have been introduced.
-
-Before:
-```yaml
-b.e[+].name: Mike Farah
-```
-
-After:
-```yaml
-- command: update 
-  path: b.e[+].thing
-  value:
-    #great 
-    things: frog # wow!
-- command: delete
-  path: b.d
-```
-
-https://github.com/mikefarah/yq/issues/305
-
-## Reading and splatting, matching results are printed once per line.
-  e.g:
-
-```json
-parent:
-  childA: 
-    no: matches here
-  childB:
-    there: matches
-    hi: no match
-    there2: also matches
-```
-
-```bash
-yq r sample.yaml 'parent.*.there*'
-```
-
-old
-```yaml
-- null
-- - matches
-  - also matches
-```
-
-new
-```yaml
-matches
-also matches
-```
-
-and you can print the matching paths:
-
-yq r --printMode pv sample.yaml 'parent.*.there*'
-
-```yaml
-parent.childB.there: matches
-parent.childB.there2: also matches
-```

action.yml

@@ -0,0 +1,13 @@
+name: 'yq - portable yaml processor'
+description: 'create, read, update, delete, merge, validate and do more with yaml'
+icon: command
+color: gray-dark
+inputs:
+  cmd:
+    description: 'The Command which should be run'
+    required: true
+runs:
+  using: 'docker'
+  image: 'github-action/Dockerfile'
+  args:
+    - ${{ inputs.cmd }}

cmd/compare.go

@@ -0,0 +1,89 @@
+package cmd
+
+import (
+	"bufio"
+	"bytes"
+	"os"
+	"strings"
+
+	"github.com/kylelemons/godebug/diff"
+	"github.com/mikefarah/yq/v3/pkg/yqlib"
+	errors "github.com/pkg/errors"
+	"github.com/spf13/cobra"
+)
+
+// turn off for unit tests :(
+var forceOsExit = true
+
+func createCompareCmd() *cobra.Command {
+	var cmdCompare = &cobra.Command{
+		Use:     "compare [yaml_file_a] [yaml_file_b]",
+		Aliases: []string{"x"},
+		Short:   "yq x [--prettyPrint/-P] dataA.yaml dataB.yaml 'b.e(name==fr*).value'",
+		Example: `
+yq x - data2.yml # reads from stdin
+yq x -pp dataA.yaml dataB.yaml '**' # compare paths
+yq x -d1 dataA.yaml dataB.yaml 'a.b.c'
+`,
+		Long: "Deeply compares two yaml files, prints the difference. Use with prettyPrint flag to ignore formatting differences.",
+		RunE: compareDocuments,
+	}
+	cmdCompare.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
+	cmdCompare.PersistentFlags().StringVarP(&printMode, "printMode", "p", "v", "print mode (v (values, default), p (paths), pv (path and value pairs)")
+	cmdCompare.PersistentFlags().StringVarP(&defaultValue, "defaultValue", "D", "", "default value printed when there are no results")
+	cmdCompare.PersistentFlags().BoolVarP(&stripComments, "stripComments", "", false, "strip comments out before comparing")
+	cmdCompare.PersistentFlags().BoolVarP(&explodeAnchors, "explodeAnchors", "X", false, "explode anchors")
+	return cmdCompare
+}
+
+func compareDocuments(cmd *cobra.Command, args []string) error {
+	var path = ""
+
+	if len(args) < 2 {
+		return errors.New("Must provide at 2 yaml files")
+	} else if len(args) > 2 {
+		path = args[2]
+	}
+
+	var updateAll, docIndexInt, errorParsingDocIndex = parseDocumentIndex()
+	if errorParsingDocIndex != nil {
+		return errorParsingDocIndex
+	}
+
+	var matchingNodesA []*yqlib.NodeContext
+	var matchingNodesB []*yqlib.NodeContext
+	var errorDoingThings error
+
+	matchingNodesA, errorDoingThings = readYamlFile(args[0], path, updateAll, docIndexInt)
+
+	if errorDoingThings != nil {
+		return errorDoingThings
+	}
+
+	matchingNodesB, errorDoingThings = readYamlFile(args[1], path, updateAll, docIndexInt)
+	if errorDoingThings != nil {
+		return errorDoingThings
+	}
+
+	var dataBufferA bytes.Buffer
+	var dataBufferB bytes.Buffer
+	errorDoingThings = printResults(matchingNodesA, bufio.NewWriter(&dataBufferA))
+	if errorDoingThings != nil {
+		return errorDoingThings
+	}
+	errorDoingThings = printResults(matchingNodesB, bufio.NewWriter(&dataBufferB))
+	if errorDoingThings != nil {
+		return errorDoingThings
+	}
+
+	diffString := diff.Diff(strings.TrimSuffix(dataBufferA.String(), "\n"), strings.TrimSuffix(dataBufferB.String(), "\n"))
+
+	if len(diffString) > 1 {
+		cmd.Print(diffString)
+		cmd.Print("\n")
+		if forceOsExit {
+			os.Exit(1)
+		}
+	}
+	return nil
+}

cmd/compare_test.go

@@ -0,0 +1,115 @@
+package cmd
+
+import (
+	"testing"
+
+	"github.com/mikefarah/yq/v3/test"
+)
+
+func TestCompareSameCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "compare ../examples/data1.yaml ../examples/data1.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := ``
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestCompareIgnoreCommentsCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "compare --stripComments ../examples/data1.yaml ../examples/data1-no-comments.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := ``
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestCompareDontIgnoreCommentsCmd(t *testing.T) {
+	forceOsExit = false
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "compare ../examples/data1.yaml ../examples/data1-no-comments.yaml")
+
+	expectedOutput := `-a: simple # just the best
++a: simple
+ b: [1, 2]
+ c:
+   test: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestCompareExplodeAnchorsCommentsCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "compare --explodeAnchors ../examples/simple-anchor.yaml ../examples/simple-anchor-exploded.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := ``
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestCompareDontExplodeAnchorsCmd(t *testing.T) {
+	forceOsExit = false
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "compare ../examples/simple-anchor.yaml ../examples/simple-anchor-exploded.yaml")
+
+	expectedOutput := `-foo: &foo
++foo:
+   a: 1
+ foobar:
+-  !!merge <<: *foo
++  a: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestCompareDifferentCmd(t *testing.T) {
+	forceOsExit = false
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "compare ../examples/data1.yaml ../examples/data3.yaml")
+
+	expectedOutput := `-a: simple # just the best
+-b: [1, 2]
++a: "simple" # just the best
++b: [1, 3]
+ c:
+   test: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestComparePrettyCmd(t *testing.T) {
+	forceOsExit = false
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "compare -P ../examples/data1.yaml ../examples/data3.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := ` a: simple # just the best
+ b:
+   - 1
+-  - 2
++  - 3
+ c:
+   test: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestComparePathsCmd(t *testing.T) {
+	forceOsExit = false
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "compare -P -ppv ../examples/data1.yaml ../examples/data3.yaml **")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := ` a: simple # just the best
+ b.[0]: 1
+-b.[1]: 2
++b.[1]: 3
+ c.test: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}

cmd/constant.go

@@ -7,12 +7,25 @@ import (
 
 var customTag = ""
 var printMode = "v"
+var printLength = false
+var unwrapScalar = true
+var customStyle = ""
+var anchorName = ""
+var makeAlias = false
+var stripComments = false
+var collectIntoArray = false
 var writeInplace = false
 var writeScript = ""
+var sourceYamlFile = ""
 var outputToJSON = false
+var exitStatus = false
+var prettyPrint = false
+var explodeAnchors = false
+var colorsEnabled = false
+var defaultValue = ""
+var indent = 2
 var overwriteFlag = false
 var autoCreateFlag = true
-var allowEmptyFlag = false
 var appendFlag = false
 var verbose = false
 var version = false

cmd/delete_test.go

@@ -0,0 +1,246 @@
+package cmd
+
+import (
+	"fmt"
+	"strings"
+	"testing"
+
+	"github.com/mikefarah/yq/v3/test"
+)
+
+func TestDeleteYamlCmd(t *testing.T) {
+	content := `a: 2
+b:
+  c: things
+  d: something else
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("delete %s b.c", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `a: 2
+b:
+  d: something else
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestDeleteDeepDoesNotExistCmd(t *testing.T) {
+	content := `a: 2`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("delete %s b.c", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `a: 2
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestDeleteSplatYaml(t *testing.T) {
+	content := `a: other
+b: [3, 4]
+c:
+  toast: leave
+  test: 1
+  tell: 1
+  tasty.taco: cool
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("delete %s c.te*", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `a: other
+b: [3, 4]
+c:
+  toast: leave
+  tasty.taco: cool
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestDeleteSplatArrayYaml(t *testing.T) {
+	content := `a: 2
+b:
+ hi:
+  - thing: item1
+    name: fred
+  - thing: item2
+    name: sam
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("delete %s b.hi[*].thing", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `a: 2
+b:
+  hi:
+    - name: fred
+    - name: sam
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestDeleteDeepSplatArrayYaml(t *testing.T) {
+	content := `thing: 123
+b:
+ hi:
+  - thing: item1
+    name: fred
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("delete %s **.thing", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `b:
+  hi:
+    - name: fred
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestDeleteSplatPrefixYaml(t *testing.T) {
+	content := `a: 2
+b:
+ hi:
+   c: things
+   d: something else
+ there:
+   c: more things
+   d: more something else
+ there2:
+   c: more things also
+   d: more something else also
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("delete %s b.there*.c", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `a: 2
+b:
+  hi:
+    c: things
+    d: something else
+  there:
+    d: more something else
+  there2:
+    d: more something else also
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestDeleteYamlArrayCmd(t *testing.T) {
+	content := `- 1
+- 2
+- 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("delete %s [1]", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `- 1
+- 3
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestDeleteYamlArrayExpressionCmd(t *testing.T) {
+	content := `- name: fred
+- name: cat
+- name: thing
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("delete %s (name==cat)", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `- name: fred
+- name: thing
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestDeleteYamlMulti(t *testing.T) {
+	content := `apples: great
+---
+- 1
+- 2
+- 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("delete -d 1 %s [1]", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+
+	expectedOutput := `apples: great
+---
+- 1
+- 3
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestDeleteYamlMultiAllCmd(t *testing.T) {
+	content := `b:
+  c: 3
+apples: great
+---
+apples: great
+something: else
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("delete %s -d * apples", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 3
+---
+something: else`
+	test.AssertResult(t, expectedOutput, strings.Trim(result.Output, "\n "))
+}

cmd/merge.go

@@ -1,11 +1,10 @@
 package cmd
 
 import (
-	"strings"
-
 	"github.com/mikefarah/yq/v3/pkg/yqlib"
 	errors "github.com/pkg/errors"
 	"github.com/spf13/cobra"
+	yaml "gopkg.in/yaml.v3"
 )
 
 func createMergeCmd() *cobra.Command {
@@ -34,27 +33,40 @@ If append flag is set then existing arrays will be merged with the arrays from e
 	cmdMerge.PersistentFlags().BoolVarP(&overwriteFlag, "overwrite", "x", false, "update the yaml file by overwriting existing values")
 	cmdMerge.PersistentFlags().BoolVarP(&autoCreateFlag, "autocreate", "c", true, "automatically create any missing entries")
 	cmdMerge.PersistentFlags().BoolVarP(&appendFlag, "append", "a", false, "update the yaml file by appending array values")
-	cmdMerge.PersistentFlags().BoolVarP(&allowEmptyFlag, "allow-empty", "e", false, "allow empty yaml files")
 	cmdMerge.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
 	return cmdMerge
 }
 
-func mergeProperties(cmd *cobra.Command, args []string) error {
+/*
-	if len(args) < 2 {
+* We don't deeply traverse arrays when appending a merge, instead we want to
-		return errors.New("Must provide at least 2 yaml files")
+* append the entire array element.
+ */
+func createReadFunctionForMerge() func(*yaml.Node) ([]*yqlib.NodeContext, error) {
+	return func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error) {
+		return lib.Get(dataBucket, "**", !appendFlag)
 	}
-	// first generate update commands from the file
+}
-	var filesToMerge = args[1:]
+
+func mergeProperties(cmd *cobra.Command, args []string) error {
 	var updateCommands []yqlib.UpdateCommand = make([]yqlib.UpdateCommand, 0)
 
-	for _, fileToMerge := range filesToMerge {
+	if len(args) < 1 {
-		matchingNodes, errorProcessingFile := readYamlFile(fileToMerge, "**", false, 0)
+		return errors.New("Must provide at least 1 yaml file")
-		if errorProcessingFile != nil && (!allowEmptyFlag || !strings.HasPrefix(errorProcessingFile.Error(), "Could not process document index")) {
+	}
-			return errorProcessingFile
+
-		}
+	if len(args) > 1 {
-		for _, matchingNode := range matchingNodes {
+		// first generate update commands from the file
-			mergePath := lib.MergePathStackToString(matchingNode.PathStack, appendFlag)
+		var filesToMerge = args[1:]
-			updateCommands = append(updateCommands, yqlib.UpdateCommand{Command: "update", Path: mergePath, Value: matchingNode.Node, Overwrite: overwriteFlag})
+
+		for _, fileToMerge := range filesToMerge {
+			matchingNodes, errorProcessingFile := doReadYamlFile(fileToMerge, createReadFunctionForMerge(), false, 0)
+			if errorProcessingFile != nil {
+				return errorProcessingFile
+			}
+			for _, matchingNode := range matchingNodes {
+				mergePath := lib.MergePathStackToString(matchingNode.PathStack, appendFlag)
+				updateCommands = append(updateCommands, yqlib.UpdateCommand{Command: "update", Path: mergePath, Value: matchingNode.Node, Overwrite: overwriteFlag})
+			}
 		}
 	}
 

cmd/merge_test.go

@@ -0,0 +1,337 @@
+package cmd
+
+import (
+	"fmt"
+	"os"
+	"runtime"
+	"testing"
+
+	"github.com/mikefarah/yq/v3/test"
+)
+
+func TestMergeCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge ../examples/data1.yaml ../examples/data2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `a: simple # just the best
+b: [1, 2]
+c:
+  test: 1
+  toast: leave
+  tell: 1
+  tasty.taco: cool
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeOneFileCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge ../examples/data1.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `a: simple # just the best
+b: [1, 2]
+c:
+  test: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeNoAutoCreateCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge -c=false ../examples/data1.yaml ../examples/data2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `a: simple # just the best
+b: [1, 2]
+c:
+  test: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeOverwriteCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge -c=false --overwrite ../examples/data1.yaml ../examples/data2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `a: other # better than the original
+b: [3, 4]
+c:
+  test: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeAppendCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge --autocreate=false --append ../examples/data1.yaml ../examples/data2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `a: simple # just the best
+b: [1, 2, 3, 4]
+c:
+  test: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeAppendArraysCmd(t *testing.T) {
+	content := `people:
+  - name: Barry
+    age: 21`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `people:
+  - name: Roger
+    age: 44`
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --append -d* %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `people:
+  - name: Barry
+    age: 21
+  - name: Roger
+    age: 44
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeOverwriteAndAppendCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge --autocreate=false --append --overwrite ../examples/data1.yaml ../examples/data2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `a: other # better than the original
+b: [1, 2, 3, 4]
+c:
+  test: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeArraysCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge --append ../examples/sample_array.yaml ../examples/sample_array_2.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `[1, 2, 3, 4, 5]
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeCmd_Multi(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge -d1 ../examples/multiple_docs_small.yaml ../examples/data1.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `a: Easy! as one two three
+---
+another:
+  document: here
+a: simple # just the best
+b:
+  - 1
+  - 2
+c:
+  test: 1
+---
+- 1
+- 2
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeYamlMultiAllCmd(t *testing.T) {
+	content := `b:
+  c: 3
+apples: green
+---
+something: else`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `apples: red
+something: good`
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge -d* %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 3
+apples: green
+something: good
+---
+something: else
+apples: red
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeSpecialCharacterKeysCmd(t *testing.T) {
+	content := ``
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `key[bracket]: value
+key.bracket: value
+key"value": value
+key'value': value
+`
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, mergeContent, result.Output)
+}
+
+func TestMergeYamlMultiAllOverwriteCmd(t *testing.T) {
+	content := `b:
+  c: 3
+apples: green
+---
+something: else`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `apples: red
+something: good`
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --overwrite -d* %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 3
+apples: red
+something: good
+---
+something: good
+apples: red
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeYamlNullMapCmd(t *testing.T) {
+	content := `b:`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `b:
+  thing: a frog
+`
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, mergeContent, result.Output)
+}
+
+func TestMergeCmd_Error(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge")
+	if result.Error == nil {
+		t.Error("Expected command to fail due to missing arg")
+	}
+	expectedOutput := `Must provide at least 1 yaml file`
+	test.AssertResult(t, expectedOutput, result.Error.Error())
+}
+
+func TestMergeCmd_ErrorUnreadableFile(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge ../examples/data1.yaml fake-unknown")
+	if result.Error == nil {
+		t.Error("Expected command to fail due to unknown file")
+	}
+	var expectedOutput string
+	if runtime.GOOS == "windows" {
+		expectedOutput = `open fake-unknown: The system cannot find the file specified.`
+	} else {
+		expectedOutput = `open fake-unknown: no such file or directory`
+	}
+	test.AssertResult(t, expectedOutput, result.Error.Error())
+}
+
+func TestMergeCmd_Inplace(t *testing.T) {
+	filename := test.WriteTempYamlFile(test.ReadTempYamlFile("../examples/data1.yaml"))
+	err := os.Chmod(filename, os.FileMode(int(0666)))
+	if err != nil {
+		t.Error(err)
+	}
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge -i %s ../examples/data2.yaml", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	info, _ := os.Stat(filename)
+	gotOutput := test.ReadTempYamlFile(filename)
+	expectedOutput := `a: simple # just the best
+b: [1, 2]
+c:
+  test: 1
+  toast: leave
+  tell: 1
+  tasty.taco: cool
+`
+	test.AssertResult(t, expectedOutput, gotOutput)
+	test.AssertResult(t, os.FileMode(int(0666)), info.Mode())
+}
+
+func TestMergeAllowEmptyTargetCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge ../examples/empty.yaml ../examples/data1.yaml")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `a: simple # just the best
+b:
+  - 1
+  - 2
+c:
+  test: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestMergeAllowEmptyMergeCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "merge ../examples/data1.yaml ../examples/empty.yaml")
+	expectedOutput := `a: simple # just the best
+b: [1, 2]
+c:
+  test: 1
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}

cmd/new.go

@@ -1,8 +1,9 @@
 package cmd
 
 import (
+	"github.com/mikefarah/yq/v3/pkg/yqlib"
+	errors "github.com/pkg/errors"
 	"github.com/spf13/cobra"
-	yaml "gopkg.in/yaml.v3"
 )
 
 func createNewCmd() *cobra.Command {
@@ -27,11 +28,19 @@ Note that you can give a create script to perform more sophisticated yaml. This
 	}
 	cmdNew.PersistentFlags().StringVarP(&writeScript, "script", "s", "", "yaml script for creating yaml")
 	cmdNew.PersistentFlags().StringVarP(&customTag, "tag", "t", "", "set yaml tag (e.g. !!int)")
+	cmdNew.PersistentFlags().StringVarP(&customStyle, "style", "", "", "formatting style of the value: single, double, folded, flow, literal, tagged")
+	cmdNew.PersistentFlags().StringVarP(&anchorName, "anchorName", "", "", "anchor name")
+	cmdNew.PersistentFlags().BoolVarP(&makeAlias, "makeAlias", "", false, "create an alias using the value as the anchor name")
 	return cmdNew
 }
 
 func newProperty(cmd *cobra.Command, args []string) error {
-	var updateCommands, updateCommandsError = readUpdateCommands(args, 2, "Must provide <path_to_update> <value>")
+	var badArgsMessage = "Must provide <path_to_update> <value>"
+	if len(args) != 2 {
+		return errors.New(badArgsMessage)
+	}
+
+	var updateCommands, updateCommandsError = readUpdateCommands(args, 2, badArgsMessage)
 	if updateCommandsError != nil {
 		return updateCommandsError
 	}
@@ -46,9 +55,6 @@ func newProperty(cmd *cobra.Command, args []string) error {
 		}
 	}
 
-	var encoder = yaml.NewEncoder(cmd.OutOrStdout())
+	var encoder = yqlib.NewYamlEncoder(cmd.OutOrStdout(), indent, colorsEnabled)
-	encoder.SetIndent(2)
+	return encoder.Encode(&newNode)
-	errorEncoding := encoder.Encode(&newNode)
-	encoder.Close()
-	return errorEncoding
 }

cmd/new_test.go

@@ -0,0 +1,101 @@
+package cmd
+
+import (
+	"testing"
+
+	"github.com/mikefarah/yq/v3/test"
+)
+
+func TestNewCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "new b.c 3")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 3
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestNewAnchorCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "new b.c 3 --anchorName=fred")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: &fred 3
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestNewAliasCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "new b.c foo --makeAlias")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: *foo
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestNewArrayCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "new b[0] 3")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  - 3
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestNewCmd_Error(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "new b.c")
+	if result.Error == nil {
+		t.Error("Expected command to fail due to missing arg")
+	}
+	expectedOutput := `Must provide <path_to_update> <value>`
+	test.AssertResult(t, expectedOutput, result.Error.Error())
+}
+
+func TestNewWithTaggedStyleCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "new b.c cat --tag=!!str --style=tagged")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: !!str cat
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestNewWithDoubleQuotedStyleCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "new b.c cat --style=double")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: "cat"
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestNewWithSingleQuotedStyleCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "new b.c cat --style=single")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 'cat'
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}

cmd/prefix_test.go

@@ -0,0 +1,189 @@
+package cmd
+
+import (
+	"fmt"
+	"runtime"
+	"strings"
+	"testing"
+
+	"github.com/mikefarah/yq/v3/test"
+)
+
+func TestPrefixCmd(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s d", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `d:
+  b:
+    c: 3
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestPrefixCmdArray(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s [+].d.[+]", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `- d:
+    - b:
+        c: 3
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestPrefixCmd_MultiLayer(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s d.e.f", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `d:
+  e:
+    f:
+      b:
+        c: 3
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestPrefixMultiCmd(t *testing.T) {
+	content := `b:
+  c: 3
+---
+apples: great
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s -d 1 d", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 3
+---
+d:
+  apples: great
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+func TestPrefixInvalidDocumentIndexCmd(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s -df d", filename))
+	if result.Error == nil {
+		t.Error("Expected command to fail due to invalid path")
+	}
+	expectedOutput := `Document index f is not a integer or *: strconv.ParseInt: parsing "f": invalid syntax`
+	test.AssertResult(t, expectedOutput, result.Error.Error())
+}
+
+func TestPrefixBadDocumentIndexCmd(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s -d 1 d", filename))
+	if result.Error == nil {
+		t.Error("Expected command to fail due to invalid path")
+	}
+	expectedOutput := `asked to process document index 1 but there are only 1 document(s)`
+	test.AssertResult(t, expectedOutput, result.Error.Error())
+}
+func TestPrefixMultiAllCmd(t *testing.T) {
+	content := `b:
+  c: 3
+---
+apples: great
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s -d * d", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `d:
+  b:
+    c: 3
+---
+d:
+  apples: great`
+	test.AssertResult(t, expectedOutput, strings.Trim(result.Output, "\n "))
+}
+
+func TestPrefixCmd_Error(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "prefix")
+	if result.Error == nil {
+		t.Error("Expected command to fail due to missing arg")
+	}
+	expectedOutput := `Must provide <filename> <prefixed_path>`
+	test.AssertResult(t, expectedOutput, result.Error.Error())
+}
+
+func TestPrefixCmd_ErrorUnreadableFile(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "prefix fake-unknown a.b")
+	if result.Error == nil {
+		t.Error("Expected command to fail due to unknown file")
+	}
+	var expectedOutput string
+	if runtime.GOOS == "windows" {
+		expectedOutput = `open fake-unknown: The system cannot find the file specified.`
+	} else {
+		expectedOutput = `open fake-unknown: no such file or directory`
+	}
+	test.AssertResult(t, expectedOutput, result.Error.Error())
+}
+
+func TestPrefixCmd_Inplace(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("prefix -i %s d", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	gotOutput := test.ReadTempYamlFile(filename)
+	expectedOutput := `d:
+  b:
+    c: 3`
+	test.AssertResult(t, expectedOutput, strings.Trim(gotOutput, "\n "))
+}

cmd/read.go

@@ -25,6 +25,13 @@ yq r -- things.yaml '--key-starting-with-dashes.blah'
 	}
 	cmdRead.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
 	cmdRead.PersistentFlags().StringVarP(&printMode, "printMode", "p", "v", "print mode (v (values, default), p (paths), pv (path and value pairs)")
+	cmdRead.PersistentFlags().StringVarP(&defaultValue, "defaultValue", "D", "", "default value printed when there are no results")
+	cmdRead.PersistentFlags().BoolVarP(&printLength, "length", "l", false, "print length of results")
+	cmdRead.PersistentFlags().BoolVarP(&collectIntoArray, "collect", "c", false, "collect results into array")
+	cmdRead.PersistentFlags().BoolVarP(&unwrapScalar, "unwrapScalar", "", true, "unwrap scalar, print the value with no quotes, colors or comments")
+	cmdRead.PersistentFlags().BoolVarP(&stripComments, "stripComments", "", false, "print yaml without any comments")
+	cmdRead.PersistentFlags().BoolVarP(&explodeAnchors, "explodeAnchors", "X", false, "explode anchors")
+	cmdRead.PersistentFlags().BoolVarP(&exitStatus, "exitStatus", "e", false, "set exit status if no matches are found")
 	return cmdRead
 }
 
@@ -44,9 +51,16 @@ func readProperty(cmd *cobra.Command, args []string) error {
 
 	matchingNodes, errorReadingStream := readYamlFile(args[0], path, updateAll, docIndexInt)
 
-	if errorReadingStream != nil {
+	if exitStatus && len(matchingNodes) == 0 {
-		return errorReadingStream
+		cmd.SilenceUsage = true
+		return errors.New("No matches found")
 	}
 
-	return printResults(matchingNodes, cmd)
+	if errorReadingStream != nil {
+		cmd.SilenceUsage = true
+		return errorReadingStream
+	}
+	out := cmd.OutOrStdout()
+
+	return printResults(matchingNodes, out)
 }

cmd/root.go

@@ -39,19 +39,22 @@ func New() *cobra.Command {
 	}
 
 	rootCmd.PersistentFlags().BoolVarP(&verbose, "verbose", "v", false, "verbose mode")
-	rootCmd.PersistentFlags().BoolVarP(&outputToJSON, "tojson", "j", false, "output as json")
+	rootCmd.PersistentFlags().BoolVarP(&outputToJSON, "tojson", "j", false, "output as json. By default it prints a json document in one line, use the prettyPrint flag to print a formatted doc.")
+	rootCmd.PersistentFlags().BoolVarP(&prettyPrint, "prettyPrint", "P", false, "pretty print")
+	rootCmd.PersistentFlags().IntVarP(&indent, "indent", "I", 2, "sets indent level for output")
 	rootCmd.Flags().BoolVarP(&version, "version", "V", false, "Print version information and quit")
+	rootCmd.PersistentFlags().BoolVarP(&colorsEnabled, "colors", "C", false, "print with colors")
 
 	rootCmd.AddCommand(
 		createReadCmd(),
+		createCompareCmd(),
 		createValidateCmd(),
 		createWriteCmd(),
 		createPrefixCmd(),
 		createDeleteCmd(),
 		createNewCmd(),
 		createMergeCmd(),
+		createBashCompletionCmd(rootCmd),
 	)
-	rootCmd.SetOutput(os.Stdout)
-
 	return rootCmd
 }

cmd/shell_completion.go

@@ -0,0 +1,57 @@
+package cmd
+
+import (
+	"fmt"
+	"os"
+
+	"github.com/spf13/cobra"
+)
+
+var shellVariant = "bash"
+
+func createBashCompletionCmd(rootCmd *cobra.Command) *cobra.Command {
+	var completionCmd = &cobra.Command{
+		Use:   "shell-completion",
+		Short: "Generates shell completion scripts",
+		Long: `To load completion for:
+bash:
+	Run	
+	. <(yq shell-completion)
+	
+	To configure your bash shell to load completions for each session add to
+	your bashrc
+	
+	# ~/.bashrc or ~/.profile
+	. <(yq shell-completion)
+
+zsh:
+	The generated completion script should be put somewhere in your $fpath named _yq
+
+powershell:
+	Users need PowerShell version 5.0 or above, which comes with Windows 10 and 
+	can be downloaded separately for Windows 7 or 8.1. They can then write the 
+	completions to a file and source this file from their PowerShell profile, 
+	which is referenced by the $Profile environment variable.
+
+fish:
+	Save the output to a fish file and add it to your completions directory.
+
+	`,
+		RunE: func(cmd *cobra.Command, args []string) error {
+			switch shellVariant {
+			case "bash", "":
+				return rootCmd.GenBashCompletion(os.Stdout)
+			case "zsh":
+				return rootCmd.GenZshCompletion(os.Stdout)
+			case "fish":
+				return rootCmd.GenFishCompletion(os.Stdout, true)
+			case "powershell":
+				return rootCmd.GenPowerShellCompletion(os.Stdout)
+			default:
+				return fmt.Errorf("Unknown variant %v", shellVariant)
+			}
+		},
+	}
+	completionCmd.PersistentFlags().StringVarP(&shellVariant, "variation", "V", "", "shell variation: bash (default), zsh, fish, powershell")
+	return completionCmd
+}

cmd/utils.go

@@ -10,11 +10,22 @@ import (
 
 	"github.com/mikefarah/yq/v3/pkg/yqlib"
 	errors "github.com/pkg/errors"
-	"github.com/spf13/cobra"
 	yaml "gopkg.in/yaml.v3"
 )
 
+type readDataFn func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error)
+
+func createReadFunction(path string) func(*yaml.Node) ([]*yqlib.NodeContext, error) {
+	return func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error) {
+		return lib.Get(dataBucket, path, true)
+	}
+}
+
 func readYamlFile(filename string, path string, updateAll bool, docIndexInt int) ([]*yqlib.NodeContext, error) {
+	return doReadYamlFile(filename, createReadFunction(path), updateAll, docIndexInt)
+}
+
+func doReadYamlFile(filename string, readFn readDataFn, updateAll bool, docIndexInt int) ([]*yqlib.NodeContext, error) {
 	var matchingNodes []*yqlib.NodeContext
 
 	var currentIndex = 0
@@ -30,7 +41,7 @@ func readYamlFile(filename string, path string, updateAll bool, docIndexInt int)
 			}
 
 			var errorParsing error
-			matchingNodes, errorParsing = appendDocument(matchingNodes, dataBucket, path, updateAll, docIndexInt, currentIndex)
+			matchingNodes, errorParsing = appendDocument(matchingNodes, dataBucket, readFn, updateAll, docIndexInt, currentIndex)
 			if errorParsing != nil {
 				return errorParsing
 			}
@@ -46,78 +57,247 @@ func readYamlFile(filename string, path string, updateAll bool, docIndexInt int)
 
 func handleEOF(updateAll bool, docIndexInt int, currentIndex int) error {
 	log.Debugf("done %v / %v", currentIndex, docIndexInt)
-	if !updateAll && currentIndex <= docIndexInt {
+	if !updateAll && currentIndex <= docIndexInt && docIndexInt != 0 {
 		return fmt.Errorf("Could not process document index %v as there are only %v document(s)", docIndex, currentIndex)
 	}
 	return nil
 }
 
-func appendDocument(originalMatchingNodes []*yqlib.NodeContext, dataBucket yaml.Node, path string, updateAll bool, docIndexInt int, currentIndex int) ([]*yqlib.NodeContext, error) {
+func appendDocument(originalMatchingNodes []*yqlib.NodeContext, dataBucket yaml.Node, readFn readDataFn, updateAll bool, docIndexInt int, currentIndex int) ([]*yqlib.NodeContext, error) {
 	log.Debugf("processing document %v - requested index %v", currentIndex, docIndexInt)
 	yqlib.DebugNode(&dataBucket)
 	if !updateAll && currentIndex != docIndexInt {
 		return originalMatchingNodes, nil
 	}
-	log.Debugf("reading %v in document %v", path, currentIndex)
+	log.Debugf("reading in document %v", currentIndex)
-	matchingNodes, errorParsing := lib.Get(&dataBucket, path)
+	matchingNodes, errorParsing := readFn(&dataBucket)
 	if errorParsing != nil {
 		return nil, errors.Wrapf(errorParsing, "Error reading path in document index %v", currentIndex)
 	}
 	return append(originalMatchingNodes, matchingNodes...), nil
 }
 
-func printValue(node *yaml.Node, cmd *cobra.Command) error {
+func lengthOf(node *yaml.Node) int {
-	if node.Kind == yaml.ScalarNode {
+	kindToCheck := node.Kind
-		cmd.Print(node.Value)
+	if node.Kind == yaml.DocumentNode && len(node.Content) == 1 {
+		log.Debugf("length of document node, calculating length of child")
+		kindToCheck = node.Content[0].Kind
+	}
+	switch kindToCheck {
+	case yaml.ScalarNode:
+		return len(node.Value)
+	case yaml.MappingNode:
+		return len(node.Content) / 2
+	default:
+		return len(node.Content)
+	}
+}
+
+// transforms node before printing, if required
+func transformNode(node *yaml.Node) *yaml.Node {
+	if printLength {
+		return &yaml.Node{Kind: yaml.ScalarNode, Value: fmt.Sprintf("%v", lengthOf(node))}
+	}
+	return node
+}
+
+func printNode(node *yaml.Node, writer io.Writer) error {
+	var encoder yqlib.Encoder
+	if node.Kind == yaml.ScalarNode && unwrapScalar && !outputToJSON {
+		return writeString(writer, node.Value+"\n")
+	}
+	if outputToJSON {
+		encoder = yqlib.NewJsonEncoder(writer, prettyPrint, indent)
+	} else {
+		encoder = yqlib.NewYamlEncoder(writer, indent, colorsEnabled)
+	}
+	return encoder.Encode(node)
+}
+
+func removeComments(matchingNodes []*yqlib.NodeContext) {
+	for _, nodeContext := range matchingNodes {
+		removeCommentOfNode(nodeContext.Node)
+	}
+}
+
+func removeCommentOfNode(node *yaml.Node) {
+	node.HeadComment = ""
+	node.LineComment = ""
+	node.FootComment = ""
+
+	for _, child := range node.Content {
+		removeCommentOfNode(child)
+	}
+}
+
+func setStyle(matchingNodes []*yqlib.NodeContext, style yaml.Style) {
+	for _, nodeContext := range matchingNodes {
+		updateStyleOfNode(nodeContext.Node, style)
+	}
+}
+
+func updateStyleOfNode(node *yaml.Node, style yaml.Style) {
+	node.Style = style
+
+	for _, child := range node.Content {
+		updateStyleOfNode(child, style)
+	}
+}
+
+func writeString(writer io.Writer, txt string) error {
+	_, errorWriting := writer.Write([]byte(txt))
+	return errorWriting
+}
+
+func setIfNotThere(node *yaml.Node, key string, value *yaml.Node) {
+	for index := 0; index < len(node.Content); index = index + 2 {
+		keyNode := node.Content[index]
+		if keyNode.Value == key {
+			return
+		}
+	}
+	// need to add it to the map
+	mapEntryKey := yaml.Node{Value: key, Kind: yaml.ScalarNode}
+	node.Content = append(node.Content, &mapEntryKey)
+	node.Content = append(node.Content, value)
+}
+
+func applyAlias(node *yaml.Node, alias *yaml.Node) {
+	for index := 0; index < len(alias.Content); index = index + 2 {
+		keyNode := alias.Content[index]
+		log.Debugf("applying alias key %v", keyNode.Value)
+		valueNode := alias.Content[index+1]
+		setIfNotThere(node, keyNode.Value, valueNode)
+	}
+}
+
+func explodeNode(node *yaml.Node) error {
+	node.Anchor = ""
+	switch node.Kind {
+	case yaml.SequenceNode, yaml.DocumentNode:
+		for index, contentNode := range node.Content {
+			log.Debugf("exploding index %v", index)
+			errorInContent := explodeNode(contentNode)
+			if errorInContent != nil {
+				return errorInContent
+			}
+		}
+		return nil
+	case yaml.AliasNode:
+		log.Debugf("its an alias!")
+		node.Kind = node.Alias.Kind
+		node.Style = node.Alias.Style
+		node.Tag = node.Alias.Tag
+		node.Content = node.Alias.Content
+		node.Value = node.Alias.Value
+		node.Alias = nil
+		return nil
+	case yaml.MappingNode:
+		for index := 0; index < len(node.Content); index = index + 2 {
+			keyNode := node.Content[index]
+			valueNode := node.Content[index+1]
+			log.Debugf("traversing %v", keyNode.Value)
+			if keyNode.Value != "<<" {
+				errorInContent := explodeNode(valueNode)
+				if errorInContent != nil {
+					return errorInContent
+				}
+			} else {
+				if valueNode.Kind == yaml.SequenceNode {
+					log.Debugf("an alias merge list!")
+					for index := len(valueNode.Content) - 1; index >= 0; index = index - 1 {
+						aliasNode := valueNode.Content[index]
+						applyAlias(node, aliasNode.Alias)
+					}
+				} else {
+					log.Debugf("an alias merge!")
+					applyAlias(node, valueNode.Alias)
+				}
+				node.Content = append(node.Content[:index], node.Content[index+2:]...)
+				//replay that index, since the array is shorter now.
+				index = index - 2
+			}
+		}
+
+		return nil
+	default:
 		return nil
 	}
+}
 
-	bufferedWriter := bufio.NewWriter(cmd.OutOrStdout())
+func explode(matchingNodes []*yqlib.NodeContext) error {
-	defer safelyFlush(bufferedWriter)
+	log.Debug("exploding nodes")
-
+	for _, nodeContext := range matchingNodes {
-	var encoder yqlib.Encoder
+		log.Debugf("exploding %v", nodeContext.Head)
-	if outputToJSON {
+		errorExplodingNode := explodeNode(nodeContext.Node)
-		encoder = yqlib.NewJsonEncoder(bufferedWriter)
+		if errorExplodingNode != nil {
-	} else {
+			return errorExplodingNode
-		encoder = yqlib.NewYamlEncoder(bufferedWriter)
+		}
-	}
-	if err := encoder.Encode(node); err != nil {
-		return err
 	}
 	return nil
 }
 
-func printResults(matchingNodes []*yqlib.NodeContext, cmd *cobra.Command) error {
+func printResults(matchingNodes []*yqlib.NodeContext, writer io.Writer) error {
-	if len(matchingNodes) == 0 {
+	if prettyPrint {
-		log.Debug("no matching results, nothing to print")
+		setStyle(matchingNodes, 0)
-		return nil
 	}
 
-	for index, mappedDoc := range matchingNodes {
+	if stripComments {
+		removeComments(matchingNodes)
+	}
+
+	//always explode anchors when printing json
+	if explodeAnchors || outputToJSON {
+		errorExploding := explode(matchingNodes)
+		if errorExploding != nil {
+			return errorExploding
+		}
+	}
+
+	bufferedWriter := bufio.NewWriter(writer)
+	defer safelyFlush(bufferedWriter)
+
+	if len(matchingNodes) == 0 {
+		log.Debug("no matching results, nothing to print")
+		if defaultValue != "" {
+			return writeString(bufferedWriter, defaultValue)
+		}
+		return nil
+	}
+	var errorWriting error
+
+	var arrayCollection = yaml.Node{Kind: yaml.SequenceNode}
+
+	for _, mappedDoc := range matchingNodes {
 		switch printMode {
 		case "p":
-			cmd.Print(lib.PathStackToString(mappedDoc.PathStack))
+			errorWriting = writeString(bufferedWriter, lib.PathStackToString(mappedDoc.PathStack)+"\n")
-			if index < len(matchingNodes)-1 {
+			if errorWriting != nil {
-				cmd.Print("\n")
+				return errorWriting
 			}
 		case "pv", "vp":
 			// put it into a node and print that.
 			var parentNode = yaml.Node{Kind: yaml.MappingNode}
 			parentNode.Content = make([]*yaml.Node, 2)
 			parentNode.Content[0] = &yaml.Node{Kind: yaml.ScalarNode, Value: lib.PathStackToString(mappedDoc.PathStack)}
-			parentNode.Content[1] = mappedDoc.Node
+			parentNode.Content[1] = transformNode(mappedDoc.Node)
-			if err := printValue(&parentNode, cmd); err != nil {
+			if collectIntoArray {
+				arrayCollection.Content = append(arrayCollection.Content, &parentNode)
+			} else if err := printNode(&parentNode, bufferedWriter); err != nil {
 				return err
 			}
 		default:
-			if err := printValue(mappedDoc.Node, cmd); err != nil {
+			if collectIntoArray {
+				arrayCollection.Content = append(arrayCollection.Content, mappedDoc.Node)
+			} else if err := printNode(transformNode(mappedDoc.Node), bufferedWriter); err != nil {
 				return err
 			}
-			// Printing our Scalars does not print a new line at the end
+		}
-			// we only want to do that if there are more values (so users can easily script extraction of values in the yaml)
+	}
-			if index < len(matchingNodes)-1 && mappedDoc.Node.Kind == yaml.ScalarNode {
+
-				cmd.Print("\n")
+	if collectIntoArray {
-			}
+		if err := printNode(transformNode(&arrayCollection), bufferedWriter); err != nil {
+			return err
 		}
 	}
 
@@ -137,6 +317,11 @@ func parseDocumentIndex() (bool, int, error) {
 
 type updateDataFn func(dataBucket *yaml.Node, currentIndex int) error
 
+func isNullDocument(dataBucket *yaml.Node) bool {
+	return dataBucket.Kind == yaml.DocumentNode && (len(dataBucket.Content) == 0 ||
+		dataBucket.Content[0].Kind == yaml.ScalarNode && dataBucket.Content[0].Tag == "!!null")
+}
+
 func mapYamlDecoder(updateData updateDataFn, encoder yqlib.Encoder) yamlDecoderFn {
 	return func(decoder *yaml.Decoder) error {
 		var dataBucket yaml.Node
@@ -154,7 +339,15 @@ func mapYamlDecoder(updateData updateDataFn, encoder yqlib.Encoder) yamlDecoderF
 			log.Debugf("Read doc %v", currentIndex)
 			errorReading = decoder.Decode(&dataBucket)
 
-			if errorReading == io.EOF {
+			if errorReading == io.EOF && docIndexInt == 0 && currentIndex == 0 {
+				//empty document, lets just make one
+				dataBucket = yaml.Node{Kind: yaml.DocumentNode, Content: make([]*yaml.Node, 1)}
+				child := yaml.Node{Kind: yaml.MappingNode}
+				dataBucket.Content[0] = &child
+			} else if isNullDocument(&dataBucket) && (updateAll || docIndexInt == currentIndex) {
+				child := yaml.Node{Kind: yaml.MappingNode}
+				dataBucket.Content[0] = &child
+			} else if errorReading == io.EOF {
 				if !updateAll && currentIndex <= docIndexInt {
 					return fmt.Errorf("asked to process document index %v but there are only %v document(s)", docIndex, currentIndex)
 				}
@@ -167,6 +360,10 @@ func mapYamlDecoder(updateData updateDataFn, encoder yqlib.Encoder) yamlDecoderF
 				return errors.Wrapf(errorUpdating, "Error updating document at index %v", currentIndex)
 			}
 
+			if prettyPrint {
+				updateStyleOfNode(&dataBucket, 0)
+			}
+
 			errorWriting = encoder.Encode(&dataBucket)
 
 			if errorWriting != nil {
@@ -220,11 +417,22 @@ func updateDoc(inputFile string, updateCommands []yqlib.UpdateCommand, writer io
 func readAndUpdate(stdOut io.Writer, inputFile string, updateData updateDataFn) error {
 	var destination io.Writer
 	var destinationName string
+	var completedSuccessfully = false
 	if writeInplace {
 		info, err := os.Stat(inputFile)
 		if err != nil {
 			return err
 		}
+		// mkdir temp dir as some docker images does not have temp dir
+		_, err = os.Stat(os.TempDir())
+		if os.IsNotExist(err) {
+			err = os.Mkdir(os.TempDir(), 0700)
+			if err != nil {
+				return err
+			}
+		} else if err != nil {
+			return err
+		}
 		tempFile, err := ioutil.TempFile("", "temp")
 		if err != nil {
 			return err
@@ -237,7 +445,9 @@ func readAndUpdate(stdOut io.Writer, inputFile string, updateData updateDataFn)
 		destination = tempFile
 		defer func() {
 			safelyCloseFile(tempFile)
-			safelyRenameFile(tempFile.Name(), inputFile)
+			if completedSuccessfully {
+				safelyRenameFile(tempFile.Name(), inputFile)
+			}
 		}()
 	} else {
 		destination = stdOut
@@ -251,11 +461,14 @@ func readAndUpdate(stdOut io.Writer, inputFile string, updateData updateDataFn)
 
 	var encoder yqlib.Encoder
 	if outputToJSON {
-		encoder = yqlib.NewJsonEncoder(bufferedWriter)
+		encoder = yqlib.NewJsonEncoder(bufferedWriter, prettyPrint, indent)
 	} else {
-		encoder = yqlib.NewYamlEncoder(bufferedWriter)
+		encoder = yqlib.NewYamlEncoder(bufferedWriter, indent, colorsEnabled)
 	}
-	return readStream(inputFile, mapYamlDecoder(updateData, encoder))
+
+	var errorProcessing = readStream(inputFile, mapYamlDecoder(updateData, encoder))
+	completedSuccessfully = errorProcessing == nil
+	return errorProcessing
 }
 
 type updateCommandParsed struct {
@@ -283,14 +496,30 @@ func readUpdateCommands(args []string, expectedArgs int, badArgsMessage string)
 		}
 
 		log.Debugf("Read write commands file '%v'", updateCommands)
-	} else if len(args) < expectedArgs {
+	} else if sourceYamlFile != "" && len(args) == expectedArgs-1 {
-		return nil, errors.New(badArgsMessage)
+		log.Debugf("Reading value from %v", sourceYamlFile)
-	} else {
+		var value yaml.Node
+		err := readData(sourceYamlFile, 0, &value)
+		if err != nil && err != io.EOF {
+			return nil, err
+		}
+		log.Debug("args %v", args[expectedArgs-2])
+		updateCommands = make([]yqlib.UpdateCommand, 1)
+		updateCommands[0] = yqlib.UpdateCommand{Command: "update", Path: args[expectedArgs-2], Value: value.Content[0], Overwrite: true}
+	} else if len(args) == expectedArgs {
 		updateCommands = make([]yqlib.UpdateCommand, 1)
 		log.Debug("args %v", args)
 		log.Debug("path %v", args[expectedArgs-2])
 		log.Debug("Value %v", args[expectedArgs-1])
-		updateCommands[0] = yqlib.UpdateCommand{Command: "update", Path: args[expectedArgs-2], Value: valueParser.Parse(args[expectedArgs-1], customTag), Overwrite: true}
+		updateCommands[0] = yqlib.UpdateCommand{Command: "update", Path: args[expectedArgs-2], Value: valueParser.Parse(args[expectedArgs-1], customTag, customStyle, anchorName, makeAlias), Overwrite: true}
+	} else if len(args) == expectedArgs-1 {
+		// don't update the value
+		updateCommands = make([]yqlib.UpdateCommand, 1)
+		log.Debug("args %v", args)
+		log.Debug("path %v", args[expectedArgs-2])
+		updateCommands[0] = yqlib.UpdateCommand{Command: "update", Path: args[expectedArgs-2], Value: valueParser.Parse("", customTag, customStyle, anchorName, makeAlias), Overwrite: true, DontUpdateNodeValue: true}
+	} else {
+		return nil, errors.New(badArgsMessage)
 	}
 	return updateCommands, nil
 }

cmd/validate_test.go

@@ -0,0 +1,31 @@
+package cmd
+
+import (
+	"fmt"
+	"testing"
+
+	"github.com/mikefarah/yq/v3/test"
+)
+
+func TestValidateCmd(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "validate ../examples/sample.yaml b.c")
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, "", result.Output)
+}
+
+func TestValidateBadDataCmd(t *testing.T) {
+	content := `[!Whatever]`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("validate %s", filename))
+	if result.Error == nil {
+		t.Error("Expected command to fail")
+	}
+	expectedOutput := `yaml: line 1: did not find expected ',' or ']'`
+	test.AssertResult(t, expectedOutput, result.Error.Error())
+}

cmd/version.go

@@ -11,7 +11,7 @@ var (
 	GitDescribe string
 
 	// Version is main version number that is being run at the moment.
-	Version = "3.0.1"
+	Version = "3.3.2"
 
 	// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
 	// then it means that it is a final release. Otherwise, this is a pre-release

cmd/write.go

@@ -43,8 +43,12 @@ format is list of update commands (update or delete) like so:
 	}
 	cmdWrite.PersistentFlags().BoolVarP(&writeInplace, "inplace", "i", false, "update the yaml file inplace")
 	cmdWrite.PersistentFlags().StringVarP(&writeScript, "script", "s", "", "yaml script for updating yaml")
+	cmdWrite.PersistentFlags().StringVarP(&sourceYamlFile, "from", "f", "", "yaml file for updating yaml (as-is)")
 	cmdWrite.PersistentFlags().StringVarP(&customTag, "tag", "t", "", "set yaml tag (e.g. !!int)")
 	cmdWrite.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
+	cmdWrite.PersistentFlags().StringVarP(&customStyle, "style", "", "", "formatting style of the value: single, double, folded, flow, literal, tagged")
+	cmdWrite.PersistentFlags().StringVarP(&anchorName, "anchorName", "", "", "anchor name")
+	cmdWrite.PersistentFlags().BoolVarP(&makeAlias, "makeAlias", "", false, "create an alias using the value as the anchor name")
 	return cmdWrite
 }
 

cmd/write_test.go

@@ -0,0 +1,593 @@
+package cmd
+
+import (
+	"fmt"
+	"runtime"
+	"strings"
+	"testing"
+
+	"github.com/mikefarah/yq/v3/test"
+)
+
+func TestWriteCmd(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c 7", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 7
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteWithTaggedStyleCmd(t *testing.T) {
+	content := `b:
+  c: dog
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c cat --tag=!!str --style=tagged", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: !!str cat
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteWithDoubleQuotedStyleCmd(t *testing.T) {
+	content := `b:
+  c: dog
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c cat --style=double", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: "cat"
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteUpdateStyleOnlyCmd(t *testing.T) {
+	content := `b:
+  c: dog
+  d: things
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.* --style=single", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 'dog'
+  d: 'things'
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteUpdateTagOnlyCmd(t *testing.T) {
+	content := `b:
+  c: true
+  d: false
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.* --tag=!!str", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: "true"
+  d: "false"
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteWithSingleQuotedStyleCmd(t *testing.T) {
+	content := `b:
+  c: dog
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c cat --style=single", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 'cat'
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteWithLiteralStyleCmd(t *testing.T) {
+	content := `b:
+  c: dog
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c cat --style=literal", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: |-
+    cat
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteWithFoldedStyleCmd(t *testing.T) {
+	content := `b:
+  c: dog
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c cat --style=folded", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: >-
+    cat
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteEmptyMultiDocCmd(t *testing.T) {
+	content := `# this is empty
+---
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s c 7", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `c: 7
+
+# this is empty
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteSurroundingEmptyMultiDocCmd(t *testing.T) {
+	content := `---
+# empty
+---
+cat: frog
+---
+
+# empty
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s -d1 c 7", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `
+
+# empty
+---
+cat: frog
+c: 7
+---
+
+
+# empty
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteFromFileCmd(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	source := `kittens: are cute # sure are!`
+	fromFilename := test.WriteTempYamlFile(source)
+	defer test.RemoveTempYamlFile(fromFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c -f %s", filename, fromFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c:
+    kittens: are cute # sure are!
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteEmptyCmd(t *testing.T) {
+	content := ``
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c 7", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 7
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteAutoCreateCmd(t *testing.T) {
+	content := `applications:
+  - name: app
+    env:`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s applications[0].env.hello world", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `applications:
+  - name: app
+    env:
+      hello: world
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteCmdScript(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	updateScript := `- command: update
+  path: b.c
+  value: 7`
+	scriptFilename := test.WriteTempYamlFile(updateScript)
+	defer test.RemoveTempYamlFile(scriptFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write --script %s %s", scriptFilename, filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 7
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteCmdEmptyScript(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	updateScript := ``
+	scriptFilename := test.WriteTempYamlFile(updateScript)
+	defer test.RemoveTempYamlFile(scriptFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write --script %s %s", scriptFilename, filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 3
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteMultiCmd(t *testing.T) {
+	content := `b:
+  c: 3
+---
+apples: great
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s -d 1 apples ok", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 3
+---
+apples: ok
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+func TestWriteInvalidDocumentIndexCmd(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s -df apples ok", filename))
+	if result.Error == nil {
+		t.Error("Expected command to fail due to invalid path")
+	}
+	expectedOutput := `Document index f is not a integer or *: strconv.ParseInt: parsing "f": invalid syntax`
+	test.AssertResult(t, expectedOutput, result.Error.Error())
+}
+
+func TestWriteBadDocumentIndexCmd(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s -d 1 apples ok", filename))
+	if result.Error == nil {
+		t.Error("Expected command to fail due to invalid path")
+	}
+	expectedOutput := `asked to process document index 1 but there are only 1 document(s)`
+	test.AssertResult(t, expectedOutput, result.Error.Error())
+}
+func TestWriteMultiAllCmd(t *testing.T) {
+	content := `b:
+  c: 3
+---
+apples: great
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s -d * apples ok", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: 3
+apples: ok
+---
+apples: ok`
+	test.AssertResult(t, expectedOutput, strings.Trim(result.Output, "\n "))
+}
+
+func TestWriteCmd_EmptyArray(t *testing.T) {
+	content := `b: 3`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s a []", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b: 3
+a: []
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteCmd_Error(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "write")
+	if result.Error == nil {
+		t.Error("Expected command to fail due to missing arg")
+	}
+	expectedOutput := `Must provide <filename> <path_to_update> <value>`
+	test.AssertResult(t, expectedOutput, result.Error.Error())
+}
+
+func TestWriteCmd_ErrorUnreadableFile(t *testing.T) {
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, "write fake-unknown a.b 3")
+	if result.Error == nil {
+		t.Error("Expected command to fail due to unknown file")
+	}
+	var expectedOutput string
+	if runtime.GOOS == "windows" {
+		expectedOutput = `open fake-unknown: The system cannot find the file specified.`
+	} else {
+		expectedOutput = `open fake-unknown: no such file or directory`
+	}
+	test.AssertResult(t, expectedOutput, result.Error.Error())
+}
+
+func TestWriteCmd_Inplace(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write -i %s b.c 7", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	gotOutput := test.ReadTempYamlFile(filename)
+	expectedOutput := `b:
+  c: 7`
+	test.AssertResult(t, expectedOutput, strings.Trim(gotOutput, "\n "))
+}
+
+func TestWriteCmd_InplaceError(t *testing.T) {
+	content := `b: cat
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write -i %s b.c 7", filename))
+	if result.Error == nil {
+		t.Error("Expected Error to occur!")
+	}
+	gotOutput := test.ReadTempYamlFile(filename)
+	test.AssertResult(t, content, gotOutput)
+}
+
+func TestWriteCmd_Append(t *testing.T) {
+	content := `b:
+  - foo
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b[+] 7", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  - foo
+  - 7
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteCmd_AppendInline(t *testing.T) {
+	content := `b: [foo]`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b[+] 7", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b: [foo, 7]
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteCmd_AppendInlinePretty(t *testing.T) {
+	content := `b: [foo]`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s -P b[+] 7", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  - foo
+  - 7
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteCmd_AppendEmptyArray(t *testing.T) {
+	content := `a: 2
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b[+] v", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `a: 2
+b:
+  - v
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteCmd_SplatArray(t *testing.T) {
+	content := `b:
+- c: thing
+- c: another thing
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b[*].c new", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  - c: new
+  - c: new
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteCmd_SplatMap(t *testing.T) {
+	content := `b:
+  c: thing
+  d: another thing
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.* new", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: new
+  d: new
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteCmd_SplatMapEmpty(t *testing.T) {
+	content := `b:
+  c: thing
+  d: another thing
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c.* new", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c: {}
+  d: another thing
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}

commands/compare.md

@@ -0,0 +1,103 @@
+---
+description: Deeply compare two yaml documents
+---
+
+# Compare
+
+```bash
+yq compare <yaml_file_1> <yaml_file_2> <path_expression>
+```
+
+Compares the matching yaml nodes at path expression in the two yaml documents. See [path expression](../usage/path-expressions.md) for more details. Difference calculated line by line, and is printed out line by line where the first character of each line is either:
+
+*  `` a space, indicating no change at this line
+* `-` a minus ,indicating the line is not present in the second document \(it's removed\)
+* `+` a plus, indicating that the line is not present in the first document \(it's added\)
+
+If there are differences then `yq` will print out the differences and exit with code 1. If there are no differences, then nothing will be printed and the exit code will be 0.
+
+## Example data
+
+Given data1.yaml
+
+```yaml
+"apples": are nice
+somethingElse: cool # this is nice
+favouriteNumbers: [1,2,3]
+noDifference: it's the same
+```
+
+and data2.yaml
+
+```yaml
+apples: are nice
+somethingElse: cool # yeah i like it
+favouriteNumbers:
+- 1
+- 3
+- 4
+noDifference: it's the same
+```
+
+## Basic
+
+Basic will compare the yaml documents 'as-is'
+
+```yaml
+yq compare data1.yaml data2.yaml
+```
+
+yields
+
+```text
+-"apples": are nice
+-somethingElse: cool # this is nice
+-favouriteNumbers: [1, 2, 3]
++apples: are nice
++somethingElse: cool # yeah i like it
++favouriteNumbers:
++- 1
++- 3
++- 4
+ noDifference: it's the same
+```
+
+## Formatted
+
+Most of the time, it will make sense to [format](../usage/output-format.md#pretty-print) the documents before comparing:
+
+```text
+yq compare --prettyPrint data1.yaml data2.yml
+```
+
+yields
+
+```text
+ apples: are nice
+-somethingElse: cool # this is nice
++somethingElse: cool # yeah i like it
+ favouriteNumbers:
+ - 1
+-- 2
+ - 3
++- 4
+ noDifference: it's the same
+```
+
+## Using path expressions
+
+Use [path expressions](../usage/path-expressions.md) to compare subsets of yaml documents
+
+```text
+yq compare -P data1.yaml data2.yml favouriteNumbers
+```
+
+yields
+
+```text
+ - 1
+-- 2
+ - 3
++- 4
+```
+

commands/create.md

@@ -0,0 +1,58 @@
+# Create
+
+```text
+yq n <path_expression> <new value>
+```
+
+This works in the same way as the write command, but you don't pass in an existing Yaml file. Currently this does not support creating multiple documents in a single yaml file.
+
+See docs for [path expression](../usage/path-expressions.md) and [value parsing](../usage/value-parsing.md) for more details, including controlling quotes and tags.
+
+## Creating a simple yaml file
+
+```bash
+yq n b.c cat
+```
+
+will output:
+
+```yaml
+b:
+  c: cat
+```
+
+## Creating using a create script
+
+Create scripts follow the same format as the update scripts.
+
+Given a script create\_instructions.yaml of:
+
+```yaml
+- command: update 
+  path: b.c
+  value:
+    #great 
+    things: frog # wow!
+```
+
+then
+
+```bash
+yq n -s create_instructions.yaml
+```
+
+will output:
+
+```yaml
+b:
+  c:
+    #great
+    things: frog # wow!
+```
+
+You can also pipe the instructions in:
+
+```bash
+cat create_instructions.yaml | yq n -s -
+```
+

commands/delete.md

@@ -0,0 +1,110 @@
+---
+description: Deletes all the matching nodes for the path expression in the given yaml input
+---
+
+# Delete
+
+```text
+yq delete <yaml_file|-> <path_expression>
+```
+
+See docs for [path expression](../usage/path-expressions.md) for more details.
+
+## Deleting from a simple document
+
+Given a sample.yaml file of:
+
+```yaml
+b:
+  c: 2
+  apples: green
+```
+
+then
+
+```bash
+yq d sample.yaml b.c
+```
+
+will output
+
+```yaml
+b:
+  apples: green
+```
+
+## From STDIN
+
+Use "-" \(without quotes\) in-place of a file name if you wish to pipe in input from STDIN.
+
+```bash
+cat sample.yaml | yq d - b.c
+```
+
+## Deleting in-place
+
+```bash
+yq d -i sample.yaml b.c
+```
+
+will update the sample.yaml file so that the 'c' node is deleted
+
+## Multiple Documents
+
+### Delete from single document
+
+Given a sample.yaml file of:
+
+```yaml
+something: else
+field: leaveMe
+---
+b:
+  c: 2
+field: deleteMe
+```
+
+then
+
+```bash
+yq w -d1 sample.yaml field
+```
+
+will output:
+
+```yaml
+something: else
+field: leaveMe
+---
+b:
+  c: 2
+```
+
+### Delete from all documents
+
+Given a sample.yaml file of:
+
+```yaml
+something: else
+field: deleteMe
+---
+b:
+  c: 2
+field: deleteMeToo
+```
+
+then
+
+```bash
+yq w -d'*' sample.yaml field
+```
+
+will output:
+
+```yaml
+something: else
+---
+b:
+  c: 2
+```
+

commands/merge.md

@@ -0,0 +1,232 @@
+---
+description: Merge multiple yaml files into a one
+---
+
+# Merge
+
+Yaml files can be merged using the 'merge' command. Each additional file merged with the first file will set values for any key not existing already or where the key has no value.
+
+```text
+yq m <yaml_file> <yaml_file2> <yaml_file3>...
+```
+
+## Merge example
+
+Given a data1.yaml file of:
+
+```yaml
+a: simple
+b: [1, 2]
+```
+
+and data2.yaml file of:
+
+```yaml
+a: other
+c:
+  test: 1
+```
+
+then
+
+```bash
+yq merge data1.yaml data2.yaml
+```
+
+will output:
+
+```yaml
+a: simple
+b: [1, 2]
+c:
+  test: 1
+```
+
+## Updating files in-place
+
+```bash
+yq m -i data1.yaml data2.yaml
+```
+
+will update the data1.yaml file with the merged result.
+
+## Overwrite values
+
+Given a data1.yaml file of:
+
+```yaml
+a: simple
+b: [1, 2]
+d: left alone
+```
+
+and data2.yaml file of:
+
+```yaml
+a: other
+b: [3, 4]
+c:
+  test: 1
+```
+
+then
+
+```bash
+yq m -x data1.yaml data2.yaml
+```
+
+will output:
+
+```yaml
+a: other
+b: [3, 4]
+c:
+  test: 1
+d: left alone
+```
+
+Notice that 'b' does not result in the merging of the values within an array.
+
+## Append values with arrays
+
+Given a data1.yaml file of:
+
+```yaml
+a: simple
+b: [1, 2]
+d: hi
+```
+
+and data2.yaml file of:
+
+```yaml
+a: something
+b: [3, 4]
+c:
+  test: 2
+  other: true
+```
+
+then
+
+```bash
+yq m -a data1.yaml data2.yaml
+```
+
+will output:
+
+```yaml
+a: simple
+b: [1, 2, 3, 4]
+c:
+  test: 2
+  other: true
+d: hi
+```
+
+Note that the 'b' array has concatenated the values from the second data file. Also note that other map keys are not overridden \(field a\).
+
+## Auto-create
+
+By default, `yq` will automatically create any missing entries in the target yaml file. This can be turned off so that only matching paths are merged in. When turned off - you will most likely want to use the [override flag](merge.md#overwrite-values).
+
+Given a data1.yml file of:
+
+```yaml
+a: thing
+b: something else
+```
+
+and data2.yml file of:
+
+```yaml
+b: new value
+d: not in original
+```
+
+Then
+
+```yaml
+yq m --overwrite --autocreate=false data1.yml data2.yml
+```
+
+will yield
+
+```yaml
+a: thing
+b: new value
+```
+
+
+
+## Multiple Documents
+
+### Merge into single document
+
+Currently yq only has multi-document support for the _first_ document being merged into. The remaining yaml files will have their first document selected.
+
+Given a data1.yaml file of:
+
+```yaml
+something: else
+---
+a: simple
+b: cat
+```
+
+and data3.yaml file of:
+
+```yaml
+b: dog
+```
+
+then
+
+```bash
+yq m -x -d1 data1.yaml data3.yaml
+```
+
+will output:
+
+```yaml
+something: else
+---
+a: simple
+b: dog
+```
+
+### Merge into all documents
+
+Currently yq only has multi-document support for the _first_ document being merged into. The remaining yaml files will have their first document selected.
+
+Given a data1.yaml file of:
+
+```yaml
+something: else
+---
+a: simple
+b: cat
+```
+
+and data3.yaml file of:
+
+```yaml
+b: dog
+```
+
+then
+
+```bash
+yq m -x -d'*' data1.yaml data3.yaml
+```
+
+will output:
+
+```yaml
+b: dog
+something: else
+---
+a: simple
+b: dog
+```
+

commands/prefix.md

@@ -0,0 +1,103 @@
+---
+description: >-
+  Prefixes a yaml document with the given path expression. The complete yaml
+  content will be nested inside the new prefix path.
+---
+
+# Prefix
+
+```text
+yq p <yaml_file> <path>
+```
+
+See docs for [path expression](../usage/path-expressions.md) for more details.
+
+## Prefix a document
+
+Given a data1.yaml file of:
+
+```yaml
+a:
+  b: [1, 2]
+```
+
+then
+
+```bash
+yq p data1.yaml c.d
+```
+
+will output:
+
+```yaml
+c:
+  d:
+    a:
+      b: [1, 2]
+```
+
+## Updating files in-place
+
+```bash
+yq p -i data1.yaml c
+```
+
+will update the data1.yaml file so that the path 'c' prefixes the document.
+
+## Multiple Documents
+
+### Prefix a single document
+
+Given a data1.yaml file of:
+
+```yaml
+something: else
+---
+a: simple
+b: cat
+```
+
+then
+
+```bash
+yq p -d1 data1.yaml c
+```
+
+will output:
+
+```yaml
+something: else
+---
+c:
+  a: simple
+  b: cat
+```
+
+### Prefix all documents
+
+Given a data1.yaml file of:
+
+```yaml
+something: else
+---
+a: simple
+b: cat
+```
+
+then
+
+```bash
+yq p -d'*' data1.yaml c
+```
+
+will output:
+
+```yaml
+c:
+  something: else
+---
+c:
+  a: simple
+  b: cat
+```
+

commands/read.md

@@ -0,0 +1,326 @@
+---
+description: Returns matching nodes/values of a path expression for a given yaml document
+---
+
+# Read
+
+```text
+yq r <yaml_file|json_file> <path_expression>
+```
+
+Returns the matching nodes of the path expression for the given yaml file \(or STDIN\).
+
+See docs for [path expression](../usage/path-expressions.md) for more details.
+
+## Basic
+
+Given a sample.yaml file of:
+
+```yaml
+b:
+  c: 2
+```
+
+then
+
+```bash
+yq r sample.yaml b.c
+```
+
+will output the value of '2'.
+
+## From Stdin
+
+Given a sample.yaml file of:
+
+```bash
+cat sample.yaml | yq r - b.c
+```
+
+will output the value of '2'.
+
+## Default values
+
+Using the `--defaultValue/-D` flag you can specify a default value to be printed when no matching nodes are found for an expression
+
+```text
+yq r sample.yaml --defaultValue frog path.not.there
+```
+
+will yield \(assuming `path.not.there` does not match any nodes\):
+
+```text
+frog
+```
+
+## Printing matching paths
+
+By default, yq will only print the value of the path expression for the yaml document. By specifying `--printMode` or `-p` you can print the matching paths.
+
+```yaml
+a:
+  thing_a: 
+    animal: cat
+  other: 
+    animal: frog
+  thing_b: 
+    vehicle: car
+```
+
+### Path Only
+
+```bash
+yq r --printMode p "a.thing*.*"
+```
+
+will print
+
+```text
+a.thing_a.animal
+a.thing_b.vehicle
+```
+
+### Path and Value
+
+```bash
+yq r --printMode pv "a.thing*.*"
+```
+
+will print
+
+```text
+a.thing_a.animal: cat
+a.thing_b.vehicle: car
+```
+
+## Collect results into an array
+
+By default, results are printed out line by line as independent matches. This is handy for both readability as well as piping into tools like `xargs`. However, if you would like to collect the matching results into an array then use the `--collect/-C` flag. This is particularly useful with the `length` flag described below.
+
+Given:
+
+```yaml
+a:
+  thing_a: 
+    animal: cat
+  other: 
+    animal: frog
+  thing_b: 
+    vehicle: car
+```
+
+```text
+yq r sample.yaml --collect a.*.animal
+```
+
+will print
+
+```text
+- cat
+- frog
+```
+
+## Printing length of the results
+
+Use the `--length/-L` flag to print the length of results. For arrays this will be the number of items, objects the number of entries and scalars the length of the value.
+
+Given
+
+```text
+animals:
+  - cats
+  - dog
+  - cheetah
+```
+
+```text
+yq r sample.yml --length animals
+```
+
+will print 
+
+```text
+3
+```
+
+### Length of filtered results
+
+By default, filtered results are printed _independently_ so you will get the length of each result printed on a separate line:
+
+```text
+yq r sample.yaml --length --printMode pv 'animals.(.==c*)'
+```
+
+```text
+animals.[0]: 4
+animals.[2]: 7
+```
+
+However, you'll often want to know the count of the filtered results - use the `--collect` flag to collect the results in the array. The length will then return the size of the array. 
+
+```text
+yq r sample.yaml --length --collect 'animals.(.==c*)' 
+```
+
+```text
+2
+```
+
+## Anchors and Aliases
+
+The read command will print out the anchors of a document and can also traverse them.
+
+Lets take a look at a simple example file:
+
+```yaml
+foo: &foo
+  a: 1
+
+foobar: *foo
+```
+
+### Printing anchors
+
+```bash
+yq r sample.yml foo
+```
+
+will print out
+
+```yaml
+&foo
+a: 1
+```
+
+Similarly,
+
+```text
+yq r sample.yml foobar
+```
+
+prints out
+
+```yaml
+*foo
+```
+
+### Traversing anchors
+
+To traverse an anchor, we need to either explicitly reference merged in values:
+
+```text
+yq r sample.yml foobar.a
+```
+
+to get
+
+```text
+1
+```
+
+or we can use deep splat to get all the values
+
+```bash
+yq r sample.yml -p pv "foobar.**"
+```
+
+prints
+
+```yaml
+foobar.a: 1
+```
+
+The same methods work for the `<<: [*blah, *thing]`anchors.
+
+### Exploding Anchors
+
+By default anchors are not exploded \(or expanded/de-referenced\) for viewing, and the yaml is shown as-is. Use the `--explodeAnchors/-X` flag to show the anchor values.
+
+Given sample.yml:
+
+```yaml
+foo: &foo
+  a: original
+  thing: coolasdf
+  thirsty: yep
+
+bar: &bar
+  b: 2
+  thing: coconut
+  c: oldbar
+
+foobarList:
+  <<: [*foo,*bar]
+  c: newbar
+```
+
+Then
+
+```text
+yq r -X sample.yml foobarList
+```
+
+yields
+
+```text
+c: newbar
+b: 2
+thing: coconut
+a: original
+thirsty: yep
+```
+
+Note that yq processes the merge anchor list in reverse order, to ensure that the last items in the list override the preceding.
+
+## Multiple Documents
+
+### Reading from a single document
+
+Given a sample.yaml file of:
+
+```yaml
+something: else
+---
+b:
+  c: 2
+```
+
+then
+
+```bash
+yq r -d1 sample.yaml b.c
+```
+
+will output the value of '2'.
+
+### Read from all documents
+
+Reading all documents will return the result as an array. This can be converted to json using the '-j' flag if desired.
+
+Given a sample.yaml file of:
+
+```yaml
+name: Fred
+age: 22
+---
+name: Stella
+age: 23
+---
+name: Android
+age: 232
+```
+
+then
+
+```bash
+yq r -d'*' sample.yaml name
+```
+
+will output:
+
+```text
+Fred
+Stella
+Android
+```
+

commands/shell-completion.md

@@ -0,0 +1,51 @@
+---
+description: >-
+  Generate a shell completion file for supported shells
+  (bash/fish/zsh/powershell)
+---
+
+# Shell Completion
+
+```bash
+yq shell-completion --variation=zsh
+```
+
+Prints to StdOut a shell completion script for zsh shell.
+
+### Bash \(default\)
+
+```bash
+yq shell-completion
+```
+
+To configure your bash shell to load completions for each session add to your bashrc
+
+```text
+# ~/.bashrc or ~/.profile
+. <(yq shell-completion)
+```
+
+### zsh
+
+```bash
+yq shell-completion --variation=zsh
+```
+
+The generated completion script should be put somewhere in your $fpath named \_yq
+
+### fish
+
+```bash
+yq shell-completion --variation=fish
+```
+
+Save the output to a '.fish' file and add it to your completions directory.
+
+### PowerShell
+
+```bash
+yq shell-completion --variation=powershell
+```
+
+Users need PowerShell version 5.0 or above, which comes with Windows 10 and can be downloaded separately for Windows 7 or 8.1. They can then write the completions to a file and source this file from their PowerShell profile, which is referenced by the $Profile environment variable.
+

commands/validate.md

@@ -0,0 +1,54 @@
+---
+description: Validate a given yaml file
+---
+
+# Validate
+
+```text
+yq v <yaml_file|->
+```
+
+Validates the given yaml file, does nothing if its valid, otherwise it will print errors to Stderr and exit with a non 0 exit code. This works like the [read command](read.md) - but does not take a path expression and does not print the yaml if it is valid.
+
+## Basic - valid
+
+```text
+yq v valid.yaml
+```
+
+This will not print anything, and finish with a successful \(0\) exit code.
+
+## Basic - invalid, from stdin
+
+```text
+echo '[1234' | yq v -
+```
+
+This will print the parsing error to stderr:
+
+```bash
+10:43:09 main [ERRO] yaml: line 1: did not find expected ',' or ']'
+```
+
+And return a error exit code \(1\)
+
+## Multiple documents
+
+Like other commands, by default the validate command will only run against the first document in the yaml file. Note that when running against other specific document indexes, _all previous documents will also be validated._
+
+### Validating a single document
+
+```bash
+yq v -d1 multidoc.yml
+```
+
+This will validate both document 0 and document 1 \(but not document 2\)
+
+### Validating all documents
+
+```bash
+yq v -d'*' multidoc.yml
+```
+
+This will validate all documents in the yaml file. Note that \* is quoted to avoid the CLI from processing the wildcard.
+

commands/write-update.md

@@ -0,0 +1,285 @@
+---
+description: >-
+  Updates all the matching nodes of path expression in a yaml file to the
+  supplied value.
+---
+
+# Write
+
+```bash
+yq w <yaml_file> <path_expression> <new value>
+```
+
+See docs for [path expression](../usage/path-expressions.md) and [value parsing](../usage/value-parsing.md) for more details, including controlling quotes and tags.
+
+## Basic
+
+Given a sample.yaml file of:
+
+```yaml
+b:
+  c: 2
+```
+
+then
+
+```bash
+yq w sample.yaml b.c cat
+```
+
+will output:
+
+```yaml
+b:
+  c: cat
+```
+
+### Updating files in-place
+
+```bash
+yq w -i sample.yaml b.c cat
+```
+
+will update the sample.yaml file so that the value of 'c' is cat.
+
+## From STDIN
+
+```bash
+cat sample.yaml | yq w - b.c blah
+```
+
+## Adding new fields
+
+Any missing fields in the path will be created on the fly.
+
+Given a sample.yaml file of:
+
+```yaml
+b:
+  c: 2
+```
+
+then
+
+```bash
+yq w sample.yaml b.d[+] "new thing"
+```
+
+will output:
+
+```yaml
+b:
+  c: cat
+  d:
+    - new thing
+```
+
+## Appending value to an array field
+
+Given a sample.yaml file of:
+
+```yaml
+b:
+  c: 2
+  d:
+    - new thing
+    - foo thing
+```
+
+then
+
+```bash
+yq w sample.yaml "b.d[+]" "bar thing"
+```
+
+will output:
+
+```yaml
+b:
+  c: cat
+  d:
+    - new thing
+    - foo thing
+    - bar thing
+```
+
+Note that the path is in quotes to avoid the square brackets being interpreted by your shell.
+
+## Multiple Documents
+
+### Update a single document
+
+Given a sample.yaml file of:
+
+```yaml
+something: else
+---
+b:
+  c: 2
+```
+
+then
+
+```bash
+yq w -d1 sample.yaml b.c 5
+```
+
+will output:
+
+```yaml
+something: else
+---
+b:
+  c: 5
+```
+
+### Update all documents
+
+Given a sample.yaml file of:
+
+```yaml
+something: else
+---
+b:
+  c: 2
+```
+
+then
+
+```bash
+yq w -d'*' sample.yaml b.c 5
+```
+
+will output:
+
+```yaml
+something: else
+b:
+  c: 5
+---
+b:
+  c: 5
+```
+
+## Writing Anchors
+
+The `---anchorName` flag can be used to set the anchor name of a node
+
+Given a sample document of:
+
+```yaml
+commonStuff:
+    flavour: vanilla
+```
+
+Then:
+
+```bash
+yq write sample.yaml commonStuff --anchorName=commonBits
+```
+
+Will yield
+
+```yaml
+commonStuff: &commonBits
+    flavour: vanilla
+```
+
+## Writing Aliases
+
+The `--makeAlias` flag can create \(or update\) a node to be an alias to an anchor.
+
+Given a sample file of:
+
+```yaml
+commonStuff: &commonBits
+    flavour: vanilla
+```
+
+Then
+
+```bash
+yq write sample.yaml --makeAnchor foo commonBits
+```
+
+Will yield:
+
+```yaml
+commonStuff: &commonBits
+    flavour: vanilla
+foo: *commonBits
+```
+
+## Updating only styles/tags without affecting values
+
+You can use the write command to update the quoting style of nodes, or their tags, without re-specifying the values. This is done by omitting the value argument:
+
+Given a sample document:
+
+```yaml
+a:
+  c: things
+  d: other things
+```
+
+Then
+
+```bash
+yq write sample.yaml --style=single a.*
+```
+
+Will yield:
+
+```yaml
+a:
+  c: 'things'
+  d: 'other things'
+```
+
+## Using a script file to update
+
+Given a sample.yaml file of:
+
+```yaml
+b:
+  d: be gone
+  c: 2
+  e:
+    - name: Billy Bob # comment over here
+```
+
+and a script update\_instructions.yaml of:
+
+```yaml
+- command: update 
+  path: b.c
+  value:
+    #great 
+    things: frog # wow!
+- command: delete
+  path: b.d
+```
+
+then
+
+```bash
+yq w -s update_instructions.yaml sample.yaml
+```
+
+will output:
+
+```yaml
+b:
+  c:
+    #great
+    things: frog # wow!
+  e:
+  - name: Billy Bob # comment over here
+```
+
+And, of course, you can pipe the instructions in using '-':
+
+```bash
+cat update_instructions.yaml | yq w -s - sample.yaml
+```
+

debian/changelog

@@ -1,3 +1,41 @@
+yq (3.3-0) focal; urgency=medium
+
+  * You can control string styles (quotes) using the new --style flag
+  * String values now always have quotes when outputting to json
+  * Negative array indices now traverse the array backwards
+  * Added a --stripComments flag to print yaml without any comments
+  * Bumped go to version 1.14
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Thu, 30 Apr 2020 20:45:44 +0200
+
+yq (3.1-2) eoan; urgency=medium
+
+  * Bug fix: yq 3 was removing empty inline-style objects and arrays (#355)
+  * Bug fix: Merge option returned different output when switching order of 
+    merging files(#347)
+  * Bug fix: Add new object to existing array object was failing in 3.1.1 (#361)
+  * Bug fix: yq 3 empty keys did not allow merging of values (#356)
+  * Bug fix: keys quoted during merge (#363)
+  * Bug fix: Correct length with wc -l (#362)
+  * Bug fix: Write to empty document removed path (#359)
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Mon, 24 Feb 2020 20:31:58 +0100
+
+yq (3.1-1) eoan; urgency=medium
+
+  * Keeps yaml comments and formatting, can specify yaml tags when updating.
+  * Handles anchors
+  * Can print out matching paths and values when splatting
+  * JSON output works for all commands
+  * Yaml files with multiple documents are printed out as one JSON 
+    document per line.
+  * Deep splat (**) to match arbitrary paths
+  * Update scripts file format has changed to be more powerful
+  * Reading and splatting, matching results are printed once per line
+  * Bugfixing
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Tue, 11 Feb 2020 22:18:24 +0100
+
 yq (2.2-1) bionic; urgency=medium
 
   * Added Windows support for the "--inplace" command flag

debian/compat

@@ -1 +1 @@
-9
+10

debian/control

@@ -1,22 +1,22 @@
 Source: yq
 Section: devel
-Priority: extra
+Priority: optional
 Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
-Build-Depends: debhelper (>= 9),
+Build-Depends: debhelper (>=10),
-               dh-golang,
+               dh-golang (>=1.34),
-               golang-1.10-go,
+               golang-1.13,
                rsync
-Standards-Version: 3.9.6
+Standards-Version: 4.1.4
 Homepage: https://github.com/mikefarah/yq.git
 Vcs-Browser: https://github.com/mikefarah/yq.git
 Vcs-Git: https://github.com/mikefarah/yq.git
+XS-Go-Import-Path: github.com/mikefarah/yq
+XSBC-Original-Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
 
 Package: yq
 Architecture: any
-Built-Using: ${misc:Built-Using}
+Depends: ${shlibs:Depends}, ${misc:Depends}
-Depends: ${shlibs:Depends},
+Description: lightweight and portable command-line YAML processor
-         ${misc:Depends}
-Description:
- a lightweight and portable command-line YAML processor
  .
- The aim of the project is to be the [jq](https://github.com/stedolan/jq) or sed of yaml files.
+ The aim of the project is to be the 
+ [jq](https://github.com/stedolan/jq) or sed of yaml files.

debian/copyright

@@ -3,5 +3,22 @@ Upstream-Name: yq
 Source: https://github.com/mikefarah/yq.git
 
 Files: *
-Copyright: 2017 Mike Farah Ltd. All rights reserved
+Copyright: 2017 Mike Farah
-License: Proprietary
+License: Expat
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+    .
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+    .
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.

debian/rules

@@ -14,46 +14,44 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-
 PROJECT := yq
 OWNER := mikefarah
 REPO := github.com
-GOVERSION := 1.10
+GOVERSION := 1.13
+
 export DH_OPTIONS
 export DH_GOPKG := ${REPO}/${OWNER}/${PROJECT}
 export GOROOT := /usr/lib/go-${GOVERSION}
 export GOPATH := ${CURDIR}/_build
 export GOBIN := ${GOPATH}/bin
 export PATH := ${GOROOT}/bin:${GOBIN}:${PATH}
-BLDPATH := $(shell dpkg-architecture -qDEB_BUILD_GNU_TYPE)
+export GOCACHE := /tmp/gocache
-SRCDIR := ${CURDIR}/_build/src/${DH_GOPKG}
+export GOFLAGS := -mod=vendor
+
+SRCDIR := ${GOPATH}/src/${DH_GOPKG}
 DESTDIR := ${CURDIR}/debian/${PROJECT}
 BINDIR := /usr/bin
 ASSETSDIR := /usr/share/${PROJECT}
 
 %:
-	dh $@ --buildsystem=golang --with=golang
+	dh $@ --builddirectory=${GOPATH} --buildsystem=golang
 
 override_dh_auto_build:
 	mkdir -p ${SRCDIR}
 	mkdir -p ${GOBIN}
 	# copy project to local srcdir to build from there
-	rsync -avz --progress --exclude=obj-${BLDPATH} --exclude=debian . $(SRCDIR)
+	rsync -avz --progress --exclude=_build --exclude=debian --exclude=tmp. --exclude=go.mod --exclude=docs . $(SRCDIR)
 	# build go code
-	(cd ${SRCDIR} && go install ./...)
+	(cd ${SRCDIR} && go install -buildmode=pie ./...)
-
+	
 override_dh_auto_test:
 	(cd ${SRCDIR} && go test -v ./...)
 
 override_dh_auto_install:
-	mkdir -p ${DESTDIR}/${BINDIR}
+	cp ${GOBIN}/yq ${DESTDIR}/${BINDIR}
-	mkdir -p ${DESTDIR}/${ASSETSDIR}
+	cp -f ${SRCDIR}/LICENSE ${DESTDIR}/${ASSETSDIR}
-	cp ${CURDIR}/_build/bin/yq ${DESTDIR}/${BINDIR}
-	cp -rf ${SRCDIR}/LICENSE ${DESTDIR}/${ASSETSDIR}
-	cp -rf ${SRCDIR}/README.md ${DESTDIR}/${PLUGINSDIR}
 	chmod a+x ${DESTDIR}/${BINDIR}/yq
 
 override_dh_auto_clean:
 	dh_clean
-	rm -rf ${CURDIR}/obj-${BLDPATH}
 	rm -rf ${CURDIR}/_build

debian/yq.dirs

@@ -0,0 +1,3 @@
+usr/bin
+usr/share/yq
+usr/share/man/man1

examples/data1-no-comments.yaml

@@ -0,0 +1,4 @@
+a: simple
+b: [1, 2]
+c:
+  test: 1

examples/data2.yaml

@@ -4,4 +4,4 @@ c:
   toast: leave
   test: 1
   tell: 1
-  taco: cool
+  tasty.taco: cool

examples/data3.yaml

@@ -1,14 +1,4 @@
-deep1:
+a: "simple" # just the best
-   hostA:
+b: [1, 3]
-      value: 1234
+c:
-   notRelevant:
+  test: 1
-      value: bananas
-   hostB:
-      value: 5678
-deep2:
-   hostC:
-      value: 1234
-   notRelevant:
-      value: bananas
-   hostD:
-      value: 5678

examples/sample2.yaml

@@ -1,2 +0,0 @@
-b:
-  c: things

examples/simple-anchor-exploded.yaml

@@ -0,0 +1,5 @@
+foo:
+  a: 1
+
+foobar: 
+  a: 1

examples/simple-anchor.yaml

@@ -1,4 +1,5 @@
 foo: &foo
   a: 1
 
-foobar: *foo
+foobar: 
+  <<: *foo

github-action/Dockerfile

@@ -0,0 +1,5 @@
+FROM mikefarah/yq:3
+
+COPY entrypoint.sh /entrypoint.sh
+
+ENTRYPOINT ["/entrypoint.sh"]

github-action/entrypoint.sh

@@ -0,0 +1,4 @@
+#!/bin/sh -l
+
+echo "$1"
+eval $1

go.mod

@@ -1,12 +1,17 @@
 module github.com/mikefarah/yq/v3
 
 require (
-	github.com/pkg/errors v0.8.1
+	github.com/fatih/color v1.9.0
-	github.com/spf13/cobra v0.0.5
+	github.com/goccy/go-yaml v1.7.5
-	golang.org/x/tools v0.0.0-20191213221258-04c2e8eff935 // indirect
+	github.com/kylelemons/godebug v1.1.0
-	gopkg.in/imdario/mergo.v0 v0.3.7 // indirect
+	github.com/mattn/go-colorable v0.1.6 // indirect
+	github.com/pkg/errors v0.9.1
+	github.com/spf13/cobra v1.0.0
+	github.com/spf13/pflag v1.0.5 // indirect
+	golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980 // indirect
+	golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 // indirect
 	gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473
-	gopkg.in/yaml.v3 v3.0.0-20191120175047-4206685974f2
+	gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c
 )
 
-go 1.13
+go 1.14

go.sum

@@ -1,62 +1,180 @@
+cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
+github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
+github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
+github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
 github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
+github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
+github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
+github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
+github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
+github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
 github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
-github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
 github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
-github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=
+github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
+github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
+github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsrgA7czyZG/E6dU=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
+github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
+github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
+github.com/fatih/color v1.9.0 h1:8xPHl4/q1VyqGIPif1F+1V3Y3lSmrq01EabUW3CoW5s=
+github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
+github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
+github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
+github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
+github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
+github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
+github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
+github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
+github.com/goccy/go-yaml v1.7.5 h1:dWvj+p3BG11S/GlUzwzt1WZz0lhBTzTIDtmXT/ZOaPY=
+github.com/goccy/go-yaml v1.7.5/go.mod h1:wS4gNoLalDSJxo/SpngzPQ2BN4uuZVLCmbM4S3vd4+Y=
+github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
+github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
+github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
+github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
+github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
+github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
+github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
+github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
+github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
+github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
 github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
 github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
+github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
+github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
+github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
+github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
+github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
+github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
+github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
+github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
+github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
 github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
-github.com/mikefarah/yaml v2.1.0+incompatible h1:nu2cqmzk4WlWJNgnevY88faMcdrDzYGcsUjYFxEpB7Y=
+github.com/mattn/go-colorable v0.1.4 h1:snbPLB8fVfU9iwbbo30TPtbLRzwWu6aJS6Xh4eaaviA=
-github.com/mikefarah/yaml/v2 v2.4.0 h1:eYqfooY0BnvKTJxr7+ABJs13n3dg9n347GScDaU2Lww=
+github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
-github.com/mikefarah/yaml/v2 v2.4.0/go.mod h1:ahVqZF4n1W4NqwvVnZzC4es67xsW9uR/RRf2RRxieJU=
+github.com/mattn/go-colorable v0.1.6 h1:6Su7aK7lXmJ/U79bYtBjLNaha4Fs1Rg9plHpcH+vvnE=
-github.com/mikefarah/yq v2.4.0+incompatible h1:oBxbWy8R9hI3BIUUxEf0CzikWa2AgnGrGhvGQt5jgjk=
+github.com/mattn/go-colorable v0.1.6/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=
-github.com/mikefarah/yq/v2 v2.4.1 h1:tajDonaFK6WqitSZExB6fKlWQy/yCkptqxh2AXEe3N4=
+github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
+github.com/mattn/go-isatty v0.0.10/go.mod h1:qgIWMr58cqv1PHHyhnkY9lrL7etaEgOFcMEpPG5Rm84=
+github.com/mattn/go-isatty v0.0.11 h1:FxPOTFNqGkuDUGi3H/qkUbQO4ZiBa2brKq5r0l8TGeM=
+github.com/mattn/go-isatty v0.0.11/go.mod h1:PhnuNfih5lzO57/f3n+odYbM4JtupLOxQOAqxQCu2WE=
+github.com/mattn/go-isatty v0.0.12 h1:wuysRhFDzyxgEmMf5xjvJ2M9dZoWAXNNr5LSBS7uHXY=
+github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=
+github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
 github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
 github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
+github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
+github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
 github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
-github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
+github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
-github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
+github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
+github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
+github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
+github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
+github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
+github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
+github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
+github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
+github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
+github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
+github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
+github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
+github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
+github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
+github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
+github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
 github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
 github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
-github.com/spf13/cobra v0.0.5 h1:f0B+LkLX6DtmRH1isoNA9VTtNUK9K8xYd28JNNfOv/s=
+github.com/spf13/cobra v1.0.0 h1:6m/oheQuQ13N9ks4hubMG6BnvwOeaJrqSPLahSnczz8=
-github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
+github.com/spf13/cobra v1.0.0/go.mod h1:/6GTrnGXV9HjY+aR4k0oJ5tcvakLuG6EuKReYlHNrgE=
 github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
 github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
 github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
-github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
+github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
+github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/spf13/viper v1.4.0/go.mod h1:PTJ7Z/lr49W6bUbkmS1V3by4uWynFiR9p7+dSq/yZzE=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
-github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
+github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
+github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
 github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
-golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
+go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
+go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
+go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
+go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
+golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
-golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
-golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
+golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
-golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
-golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
-golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
-golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
+golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
+golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20191008105621-543471e840be/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191010194322-b09406accb47/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191026070338-33540a1f6037 h1:YyJpGZS1sBuBCzLAR1VEpK193GlqGZbnPFnPV/5Rsb4=
+golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980 h1:OjiUf46hAmXblsZdnoSXsEUSKU8r1UEzcL5RVZ4gO9Y=
+golang.org/x/sys v0.0.0-20200602225109-6fdc65e7d980/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
-golang.org/x/tools v0.0.0-20191030203535-5e247c9ad0a0 h1:s5lp4ug7qHzUccgyFdjsX7OZDzHXRaePrF3B3vmUiuM=
+golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
-golang.org/x/tools v0.0.0-20191030203535-5e247c9ad0a0/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
-golang.org/x/tools v0.0.0-20191213221258-04c2e8eff935 h1:kJQZhwFzSwJS2BxboKjdZzWczQOZx8VuH7Y8hhuGUtM=
+golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
-golang.org/x/tools v0.0.0-20191213221258-04c2e8eff935/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
-golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898 h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
+golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
+google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
+google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
+google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
+gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
-gopkg.in/imdario/mergo.v0 v0.3.7 h1:QDotlIZtaO/p+Um0ok18HRTpq5i5/SAk/qprsor+9c8=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
-gopkg.in/imdario/mergo.v0 v0.3.7/go.mod h1:9qPP6AGrlC1G2PTNXko614FwGZvorN7MiBU0Eppok+U=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/go-playground/assert.v1 v1.2.1/go.mod h1:9RXL0bg/zibRAgZUYszZSwO/z8Y/a8bDuhia5mkpMnE=
+gopkg.in/go-playground/validator.v9 v9.30.0/go.mod h1:+c9/zcJMFNgbLvly1L1V+PpxWdVbfP1avr/N00E2vyQ=
 gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473 h1:6D+BvnJ/j6e222UW8s2qTSe3wGBtvo0MbVQG/c5k8RE=
 gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473/go.mod h1:N1eN2tsCx0Ydtgjl4cqmbRCsY4/+z4cYDeqwZTk6zog=
+gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
+gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
+gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v3 v3.0.0-20191120175047-4206685974f2 h1:XZx7nhd5GMaZpmDaEHFVafUZC7ya0fuo7cSJ3UCKYmM=
+gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c h1:grhR+C34yXImVGp7EzNk+DTIk+323eIUWOmEevy6bDo=
-gopkg.in/yaml.v3 v3.0.0-20191120175047-4206685974f2/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gopkg.in/yaml.v3 v3.0.0-20200605160147-a5ece683394c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=

pkg/yqlib/color_print.go

@@ -0,0 +1,61 @@
+package yqlib
+
+import (
+	"fmt"
+	"io"
+
+	"github.com/fatih/color"
+	"github.com/goccy/go-yaml/lexer"
+	"github.com/goccy/go-yaml/printer"
+)
+
+// Thanks @risentveber!
+
+const escape = "\x1b"
+
+func format(attr color.Attribute) string {
+	return fmt.Sprintf("%s[%dm", escape, attr)
+}
+
+func ColorizeAndPrint(bytes []byte, writer io.Writer) error {
+	tokens := lexer.Tokenize(string(bytes))
+	var p printer.Printer
+	p.Bool = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgHiMagenta),
+			Suffix: format(color.Reset),
+		}
+	}
+	p.Number = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgHiMagenta),
+			Suffix: format(color.Reset),
+		}
+	}
+	p.MapKey = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgCyan),
+			Suffix: format(color.Reset),
+		}
+	}
+	p.Anchor = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgHiYellow),
+			Suffix: format(color.Reset),
+		}
+	}
+	p.Alias = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgHiYellow),
+			Suffix: format(color.Reset),
+		}
+	}
+	p.String = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgGreen),
+			Suffix: format(color.Reset),
+		}
+	}
+	_, err := writer.Write([]byte(p.PrintTokens(tokens) + "\n"))
+	return err
+}

pkg/yqlib/data_navigator.go

@@ -1,13 +1,14 @@
 package yqlib
 
 import (
+	"fmt"
 	"strconv"
 
 	yaml "gopkg.in/yaml.v3"
 )
 
 type DataNavigator interface {
-	Traverse(value *yaml.Node, path []string) error
+	Traverse(value *yaml.Node, path []interface{}) error
 }
 
 type navigator struct {
@@ -20,9 +21,10 @@ func NewDataNavigator(NavigationStrategy NavigationStrategy) DataNavigator {
 	}
 }
 
-func (n *navigator) Traverse(value *yaml.Node, path []string) error {
+func (n *navigator) Traverse(value *yaml.Node, path []interface{}) error {
 	realValue := value
 	emptyArray := make([]interface{}, 0)
+	log.Debugf("Traversing path %v", pathStackToString(path))
 	if realValue.Kind == yaml.DocumentNode {
 		log.Debugf("its a document! returning the first child")
 		return n.doTraverse(value.Content[0], "", path, emptyArray)
@@ -30,12 +32,17 @@ func (n *navigator) Traverse(value *yaml.Node, path []string) error {
 	return n.doTraverse(value, "", path, emptyArray)
 }
 
-func (n *navigator) doTraverse(value *yaml.Node, head string, tail []string, pathStack []interface{}) error {
+func (n *navigator) doTraverse(value *yaml.Node, head interface{}, tail []interface{}, pathStack []interface{}) error {
+
 	log.Debug("head %v", head)
 	DebugNode(value)
+	var nodeContext = NewNodeContext(value, head, tail, pathStack)
+
 	var errorDeepSplatting error
-	if head == "**" && value.Kind != yaml.ScalarNode {
+	if head == "**" && value.Kind != yaml.ScalarNode && n.navigationStrategy.ShouldDeeplyTraverse(nodeContext) {
-		errorDeepSplatting = n.recurse(value, head, tail, pathStack)
+		if len(pathStack) == 0 || pathStack[len(pathStack)-1] != "<<" {
+			errorDeepSplatting = n.recurse(value, head, tail, pathStack)
+		}
 		// ignore errors here, we are deep splatting so we may accidently give a string key
 		// to an array sequence
 		if len(tail) > 0 {
@@ -44,66 +51,80 @@ func (n *navigator) doTraverse(value *yaml.Node, head string, tail []string, pat
 		return errorDeepSplatting
 	}
 
-	if len(tail) > 0 {
+	if len(tail) > 0 && value.Kind != yaml.ScalarNode {
 		log.Debugf("diving into %v", tail[0])
 		DebugNode(value)
 		return n.recurse(value, tail[0], tail[1:], pathStack)
 	}
-	return n.navigationStrategy.Visit(NewNodeContext(value, head, tail, pathStack))
+	return n.navigationStrategy.Visit(nodeContext)
 }
 
 func (n *navigator) getOrReplace(original *yaml.Node, expectedKind yaml.Kind) *yaml.Node {
 	if original.Kind != expectedKind {
-		log.Debug("wanted %v but it was %v, overriding", expectedKind, original.Kind)
+		log.Debug("wanted %v but it was %v, overriding", KindString(expectedKind), KindString(original.Kind))
 		return &yaml.Node{Kind: expectedKind}
 	}
 	return original
 }
 
-func (n *navigator) recurse(value *yaml.Node, head string, tail []string, pathStack []interface{}) error {
+func (n *navigator) recurse(value *yaml.Node, head interface{}, tail []interface{}, pathStack []interface{}) error {
-	log.Debug("recursing, processing %v", head)
+	log.Debug("recursing, processing %v, pathStack %v", head, pathStackToString(pathStack))
+
+	nodeContext := NewNodeContext(value, head, tail, pathStack)
+
+	if head == "**" && !n.navigationStrategy.ShouldOnlyDeeplyVisitLeaves(nodeContext) {
+		errorVisitingDeeply := n.navigationStrategy.Visit(nodeContext)
+		if errorVisitingDeeply != nil {
+			return errorVisitingDeeply
+		}
+	}
+
 	switch value.Kind {
 	case yaml.MappingNode:
 		log.Debug("its a map with %v entries", len(value.Content)/2)
-		return n.recurseMap(value, head, tail, pathStack)
+		headString := fmt.Sprintf("%v", head)
+		return n.recurseMap(value, headString, tail, pathStack)
+
 	case yaml.SequenceNode:
 		log.Debug("its a sequence of %v things!", len(value.Content))
 
-		var index, errorParsingIndex = strconv.ParseInt(head, 10, 64) // nolint
+		switch head := head.(type) {
-		if errorParsingIndex == nil {
+		case int64:
-			return n.recurseArray(value, index, head, tail, pathStack)
+			return n.recurseArray(value, head, head, tail, pathStack)
-		} else if head == "+" {
+		default:
-			return n.appendArray(value, head, tail, pathStack)
-		}
-		return n.splatArray(value, head, tail, pathStack)
 
+			if head == "+" {
+				return n.appendArray(value, head, tail, pathStack)
+			} else if len(value.Content) == 0 && head == "**" {
+				return n.navigationStrategy.Visit(nodeContext)
+			}
+			return n.splatArray(value, head, tail, pathStack)
+		}
 	case yaml.AliasNode:
 		log.Debug("its an alias!")
 		DebugNode(value.Alias)
-		if n.navigationStrategy.FollowAlias(NewNodeContext(value, head, tail, pathStack)) {
+		if n.navigationStrategy.FollowAlias(nodeContext) {
 			log.Debug("following the alias")
 			return n.recurse(value.Alias, head, tail, pathStack)
 		}
 		return nil
 	default:
-		return nil
+		return n.navigationStrategy.Visit(nodeContext)
 	}
 }
 
-func (n *navigator) recurseMap(value *yaml.Node, head string, tail []string, pathStack []interface{}) error {
+func (n *navigator) recurseMap(value *yaml.Node, head string, tail []interface{}, pathStack []interface{}) error {
 	traversedEntry := false
 	errorVisiting := n.visitMatchingEntries(value, head, tail, pathStack, func(contents []*yaml.Node, indexInMap int) error {
-		log.Debug("recurseMap: visitMatchingEntries")
+		log.Debug("recurseMap: visitMatchingEntries for %v", contents[indexInMap].Value)
 		n.navigationStrategy.DebugVisitedNodes()
 		newPathStack := append(pathStack, contents[indexInMap].Value)
-		log.Debug("appended %v", contents[indexInMap].Value)
-		n.navigationStrategy.DebugVisitedNodes()
 		log.Debug("should I traverse? head: %v, path: %v", head, pathStackToString(newPathStack))
 		DebugNode(value)
 		if n.navigationStrategy.ShouldTraverse(NewNodeContext(contents[indexInMap+1], head, tail, newPathStack), contents[indexInMap].Value) {
 			log.Debug("recurseMap: Going to traverse")
 			traversedEntry = true
-			// contents[indexInMap+1] = n.getOrReplace(contents[indexInMap+1], guessKind(head, tail, contents[indexInMap+1].Kind))
+			contents[indexInMap+1] = n.getOrReplace(contents[indexInMap+1], guessKind(head, tail, contents[indexInMap+1].Kind))
 			errorTraversing := n.doTraverse(contents[indexInMap+1], head, tail, newPathStack)
 			log.Debug("recurseMap: Finished traversing")
 			n.navigationStrategy.DebugVisitedNodes()
@@ -118,22 +139,33 @@ func (n *navigator) recurseMap(value *yaml.Node, head string, tail []string, pat
 		return errorVisiting
 	}
 
-	if traversedEntry || n.navigationStrategy.GetPathParser().IsPathExpression(head) || !n.navigationStrategy.AutoCreateMap(NewNodeContext(value, head, tail, pathStack)) {
+	if len(value.Content) == 0 && head == "**" {
+		return n.navigationStrategy.Visit(NewNodeContext(value, head, tail, pathStack))
+	} else if traversedEntry || n.navigationStrategy.GetPathParser().IsPathExpression(head) || !n.navigationStrategy.AutoCreateMap(NewNodeContext(value, head, tail, pathStack)) {
 		return nil
 	}
 
+	_, errorParsingInt := strconv.ParseInt(head, 10, 64)
+
 	mapEntryKey := yaml.Node{Value: head, Kind: yaml.ScalarNode}
+
+	if errorParsingInt == nil {
+		// fixes a json encoding problem where keys that look like numbers
+		// get treated as numbers and cannot be used in a json map
+		mapEntryKey.Style = yaml.LiteralStyle
+	}
+
 	value.Content = append(value.Content, &mapEntryKey)
 	mapEntryValue := yaml.Node{Kind: guessKind(head, tail, 0)}
 	value.Content = append(value.Content, &mapEntryValue)
-	log.Debug("adding new node %v", head)
+	log.Debug("adding a new node %v - def a string", head)
 	return n.doTraverse(&mapEntryValue, head, tail, append(pathStack, head))
 }
 
 // need to pass the node in, as it may be aliased
 type mapVisitorFn func(contents []*yaml.Node, index int) error
 
-func (n *navigator) visitDirectMatchingEntries(node *yaml.Node, head string, tail []string, pathStack []interface{}, visit mapVisitorFn) error {
+func (n *navigator) visitDirectMatchingEntries(node *yaml.Node, head string, tail []interface{}, pathStack []interface{}, visit mapVisitorFn) error {
 	var contents = node.Content
 	for index := 0; index < len(contents); index = index + 2 {
 		content := contents[index]
@@ -148,7 +180,7 @@ func (n *navigator) visitDirectMatchingEntries(node *yaml.Node, head string, tai
 	return nil
 }
 
-func (n *navigator) visitMatchingEntries(node *yaml.Node, head string, tail []string, pathStack []interface{}, visit mapVisitorFn) error {
+func (n *navigator) visitMatchingEntries(node *yaml.Node, head string, tail []interface{}, pathStack []interface{}, visit mapVisitorFn) error {
 	var contents = node.Content
 	log.Debug("visitMatchingEntries %v", head)
 	DebugNode(node)
@@ -164,17 +196,17 @@ func (n *navigator) visitMatchingEntries(node *yaml.Node, head string, tail []st
 	return n.visitAliases(contents, head, tail, pathStack, visit)
 }
 
-func (n *navigator) visitAliases(contents []*yaml.Node, head string, tail []string, pathStack []interface{}, visit mapVisitorFn) error {
+func (n *navigator) visitAliases(contents []*yaml.Node, head string, tail []interface{}, pathStack []interface{}, visit mapVisitorFn) error {
 	// merge aliases are defined first, but we only want to traverse them
 	// if we don't find a match on this node first.
 	// traverse them backwards so that the last alias overrides the preceding.
 	// a node can either be
 	// an alias to one other node (e.g. <<: *blah)
 	// or a sequence of aliases   (e.g. <<: [*blah, *foo])
-	log.Debug("checking for aliases")
+	log.Debug("checking for aliases, head: %v, pathstack: %v", head, pathStackToString(pathStack))
 	for index := len(contents) - 2; index >= 0; index = index - 2 {
 
-		if contents[index+1].Kind == yaml.AliasNode {
+		if contents[index+1].Kind == yaml.AliasNode && contents[index].Value == "<<" {
 			valueNode := contents[index+1]
 			log.Debug("found an alias")
 			DebugNode(contents[index])
@@ -195,7 +227,7 @@ func (n *navigator) visitAliases(contents []*yaml.Node, head string, tail []stri
 	return nil
 }
 
-func (n *navigator) visitAliasSequence(possibleAliasArray []*yaml.Node, head string, tail []string, pathStack []interface{}, visit mapVisitorFn) error {
+func (n *navigator) visitAliasSequence(possibleAliasArray []*yaml.Node, head string, tail []interface{}, pathStack []interface{}, visit mapVisitorFn) error {
 	// need to search this backwards too, so that aliases defined last override the preceding.
 	for aliasIndex := len(possibleAliasArray) - 1; aliasIndex >= 0; aliasIndex = aliasIndex - 1 {
 		child := possibleAliasArray[aliasIndex]
@@ -211,7 +243,7 @@ func (n *navigator) visitAliasSequence(possibleAliasArray []*yaml.Node, head str
 	return nil
 }
 
-func (n *navigator) splatArray(value *yaml.Node, head string, tail []string, pathStack []interface{}) error {
+func (n *navigator) splatArray(value *yaml.Node, head interface{}, tail []interface{}, pathStack []interface{}) error {
 	for index, childValue := range value.Content {
 		log.Debug("processing")
 		DebugNode(childValue)
@@ -219,6 +251,9 @@ func (n *navigator) splatArray(value *yaml.Node, head string, tail []string, pat
 
 		newPathStack := append(pathStack, index)
 		if n.navigationStrategy.ShouldTraverse(NewNodeContext(childValue, head, tail, newPathStack), childValue.Value) {
+			// here we should not deeply traverse the array if we are appending..not sure how to do that.
+			// need to visit instead...
+			// easiest way is to pop off the head and pass the rest of the tail in.
 			var err = n.doTraverse(childValue, head, tail, newPathStack)
 			if err != nil {
 				return err
@@ -228,19 +263,30 @@ func (n *navigator) splatArray(value *yaml.Node, head string, tail []string, pat
 	return nil
 }
 
-func (n *navigator) appendArray(value *yaml.Node, head string, tail []string, pathStack []interface{}) error {
+func (n *navigator) appendArray(value *yaml.Node, head interface{}, tail []interface{}, pathStack []interface{}) error {
 	var newNode = yaml.Node{Kind: guessKind(head, tail, 0)}
 	value.Content = append(value.Content, &newNode)
 	log.Debug("appending a new node, %v", value.Content)
 	return n.doTraverse(&newNode, head, tail, append(pathStack, len(value.Content)-1))
 }
 
-func (n *navigator) recurseArray(value *yaml.Node, index int64, head string, tail []string, pathStack []interface{}) error {
+func (n *navigator) recurseArray(value *yaml.Node, index int64, head interface{}, tail []interface{}, pathStack []interface{}) error {
-	for int64(len(value.Content)) <= index {
+	var contentLength = int64(len(value.Content))
+	for contentLength <= index {
 		value.Content = append(value.Content, &yaml.Node{Kind: guessKind(head, tail, 0)})
+		contentLength = int64(len(value.Content))
+	}
+	var indexToUse = index
+
+	if indexToUse < 0 {
+		indexToUse = contentLength + indexToUse
 	}
 
-	value.Content[index] = n.getOrReplace(value.Content[index], guessKind(head, tail, value.Content[index].Kind))
+	if indexToUse < 0 {
+		return fmt.Errorf("Index [%v] out of range, array size is %v", index, contentLength)
+	}
 
-	return n.doTraverse(value.Content[index], head, tail, append(pathStack, index))
+	value.Content[indexToUse] = n.getOrReplace(value.Content[indexToUse], guessKind(head, tail, value.Content[indexToUse].Kind))
+
+	return n.doTraverse(value.Content[indexToUse], head, tail, append(pathStack, index))
 }

pkg/yqlib/delete_navigation_strategy.go

@@ -1,12 +1,10 @@
 package yqlib
 
 import (
-	"strconv"
-
 	yaml "gopkg.in/yaml.v3"
 )
 
-func DeleteNavigationStrategy(pathElementToDelete string) NavigationStrategy {
+func DeleteNavigationStrategy(pathElementToDelete interface{}) NavigationStrategy {
 	parser := NewPathParser()
 	return &NavigationStrategyImpl{
 		visitedNodes: []*NodeContext{},
@@ -14,12 +12,12 @@ func DeleteNavigationStrategy(pathElementToDelete string) NavigationStrategy {
 		followAlias: func(nodeContext NodeContext) bool {
 			return false
 		},
-		autoCreateMap: func(nodeContext NodeContext) bool {
+		shouldOnlyDeeplyVisitLeaves: func(nodeContext NodeContext) bool {
-			return true
+			return false
 		},
 		visit: func(nodeContext NodeContext) error {
 			node := nodeContext.Node
-			log.Debug("need to find and delete %v in here", pathElementToDelete)
+			log.Debug("need to find and delete %v in here (%v)", pathElementToDelete, pathStackToString(nodeContext.PathStack))
 			DebugNode(node)
 			if node.Kind == yaml.SequenceNode {
 				newContent := deleteFromArray(parser, node.Content, nodeContext.PathStack, pathElementToDelete)
@@ -31,12 +29,12 @@ func DeleteNavigationStrategy(pathElementToDelete string) NavigationStrategy {
 		},
 	}
 }
-func deleteFromMap(pathParser PathParser, contents []*yaml.Node, pathStack []interface{}, pathElementToDelete string) []*yaml.Node {
+func deleteFromMap(pathParser PathParser, contents []*yaml.Node, pathStack []interface{}, pathElementToDelete interface{}) []*yaml.Node {
 	newContents := make([]*yaml.Node, 0)
 	for index := 0; index < len(contents); index = index + 2 {
 		keyNode := contents[index]
 		valueNode := contents[index+1]
-		if !pathParser.MatchesNextPathElement(NewNodeContext(keyNode, pathElementToDelete, []string{}, pathStack), keyNode.Value) {
+		if !pathParser.MatchesNextPathElement(NewNodeContext(keyNode, pathElementToDelete, make([]interface{}, 0), pathStack), keyNode.Value) {
 			log.Debug("adding node %v", keyNode.Value)
 			newContents = append(newContents, keyNode, valueNode)
 		} else {
@@ -46,21 +44,23 @@ func deleteFromMap(pathParser PathParser, contents []*yaml.Node, pathStack []int
 	return newContents
 }
 
-func deleteFromArray(pathParser PathParser, content []*yaml.Node, pathStack []interface{}, pathElementToDelete string) []*yaml.Node {
+func deleteFromArray(pathParser PathParser, content []*yaml.Node, pathStack []interface{}, pathElementToDelete interface{}) []*yaml.Node {
 
-	var indexToDelete, err = strconv.ParseInt(pathElementToDelete, 10, 64) // nolint
+	switch pathElementToDelete := pathElementToDelete.(type) {
-	if err == nil {
+	case int64:
-		return deleteIndexInArray(content, indexToDelete)
+		return deleteIndexInArray(content, pathElementToDelete)
-	}
+	default:
-	log.Debug("%v is not a numeric index, finding matching patterns", pathElementToDelete)
+		log.Debug("%v is not a numeric index, finding matching patterns", pathElementToDelete)
-	var newArray = make([]*yaml.Node, 0)
+		var newArray = make([]*yaml.Node, 0)
 
-	for _, childValue := range content {
+		for _, childValue := range content {
-		if !pathParser.MatchesNextPathElement(NewNodeContext(childValue, pathElementToDelete, []string{}, pathStack), childValue.Value) {
+			if !pathParser.MatchesNextPathElement(NewNodeContext(childValue, pathElementToDelete, make([]interface{}, 0), pathStack), childValue.Value) {
-			newArray = append(newArray, childValue)
+				newArray = append(newArray, childValue)
+			}
 		}
+		return newArray
 	}
-	return newArray
+
 }
 
 func deleteIndexInArray(content []*yaml.Node, index int64) []*yaml.Node {

pkg/yqlib/encoder.go

@@ -1,6 +1,7 @@
 package yqlib
 
 import (
+	"bytes"
 	"encoding/json"
 	"io"
 
@@ -12,25 +13,61 @@ type Encoder interface {
 }
 
 type yamlEncoder struct {
-	encoder *yaml.Encoder
+	destination io.Writer
+	indent      int
+	colorise    bool
+	firstDoc    bool
 }
 
-func NewYamlEncoder(destination io.Writer) Encoder {
+func NewYamlEncoder(destination io.Writer, indent int, colorise bool) Encoder {
-	var encoder = yaml.NewEncoder(destination)
+	if indent < 0 {
-	encoder.SetIndent(2)
+		indent = 0
-	return &yamlEncoder{encoder}
+	}
+	return &yamlEncoder{destination, indent, colorise, true}
 }
 
 func (ye *yamlEncoder) Encode(node *yaml.Node) error {
-	return ye.encoder.Encode(node)
+
+	destination := ye.destination
+	tempBuffer := bytes.NewBuffer(nil)
+	if ye.colorise {
+		destination = tempBuffer
+	}
+
+	var encoder = yaml.NewEncoder(destination)
+
+	encoder.SetIndent(ye.indent)
+	// TODO: work out if the first doc had a separator or not.
+	if ye.firstDoc {
+		ye.firstDoc = false
+	} else if _, err := destination.Write([]byte("---\n")); err != nil {
+		return err
+	}
+
+	if err := encoder.Encode(node); err != nil {
+		return err
+	}
+
+	if ye.colorise {
+		return ColorizeAndPrint(tempBuffer.Bytes(), ye.destination)
+	}
+	return nil
 }
 
 type jsonEncoder struct {
 	encoder *json.Encoder
 }
 
-func NewJsonEncoder(destination io.Writer) Encoder {
+func NewJsonEncoder(destination io.Writer, prettyPrint bool, indent int) Encoder {
 	var encoder = json.NewEncoder(destination)
+	var indentString = ""
+
+	for index := 0; index < indent; index++ {
+		indentString = indentString + " "
+	}
+	if prettyPrint {
+		encoder.SetIndent("", indentString)
+	}
 	return &jsonEncoder{encoder}
 }
 

pkg/yqlib/filter_matching_node_navigation_strategy.go

@@ -4,12 +4,6 @@ func FilterMatchingNodesNavigationStrategy(value string) NavigationStrategy {
 	return &NavigationStrategyImpl{
 		visitedNodes: []*NodeContext{},
 		pathParser:   NewPathParser(),
-		followAlias: func(nodeContext NodeContext) bool {
-			return true
-		},
-		autoCreateMap: func(nodeContext NodeContext) bool {
-			return false
-		},
 		visit: func(nodeContext NodeContext) error {
 			return nil
 		},

pkg/yqlib/lib.go

@@ -13,10 +13,28 @@ import (
 var log = logging.MustGetLogger("yq")
 
 type UpdateCommand struct {
-	Command   string
+	Command             string
-	Path      string
+	Path                string
-	Value     *yaml.Node
+	Value               *yaml.Node
-	Overwrite bool
+	Overwrite           bool
+	DontUpdateNodeValue bool
+}
+
+func KindString(kind yaml.Kind) string {
+	switch kind {
+	case yaml.ScalarNode:
+		return "ScalarNode"
+	case yaml.SequenceNode:
+		return "SequenceNode"
+	case yaml.MappingNode:
+		return "MappingNode"
+	case yaml.DocumentNode:
+		return "DocumentNode"
+	case yaml.AliasNode:
+		return "AliasNode"
+	default:
+		return "unknown!"
+	}
 }
 
 func DebugNode(value *yaml.Node) {
@@ -30,7 +48,7 @@ func DebugNode(value *yaml.Node) {
 			log.Error("Error debugging node, %v", errorEncoding.Error())
 		}
 		encoder.Close()
-		log.Debug("Tag: %v", value.Tag)
+		log.Debug("Tag: %v, Kind: %v, Anchor: %v", value.Tag, KindString(value.Kind), value.Anchor)
 		log.Debug("%v", buf.String())
 	}
 }
@@ -43,7 +61,7 @@ func mergePathStackToString(pathStack []interface{}, appendArrays bool) string {
 	var sb strings.Builder
 	for index, path := range pathStack {
 		switch path.(type) {
-		case int:
+		case int, int64:
 			if appendArrays {
 				sb.WriteString("[+]")
 			} else {
@@ -51,45 +69,71 @@ func mergePathStackToString(pathStack []interface{}, appendArrays bool) string {
 			}
 
 		default:
-			sb.WriteString(fmt.Sprintf("%v", path))
+			s := fmt.Sprintf("%v", path)
+			var _, errParsingInt = strconv.ParseInt(s, 10, 64) // nolint
+
+			hasSpecial := strings.Contains(s, ".") || strings.Contains(s, "[") || strings.Contains(s, "]") || strings.Contains(s, "\"")
+			hasDoubleQuotes := strings.Contains(s, "\"")
+			wrappingCharacterStart := "\""
+			wrappingCharacterEnd := "\""
+			if hasDoubleQuotes {
+				wrappingCharacterStart = "("
+				wrappingCharacterEnd = ")"
+			}
+			if hasSpecial || errParsingInt == nil {
+				sb.WriteString(wrappingCharacterStart)
+			}
+			sb.WriteString(s)
+			if hasSpecial || errParsingInt == nil {
+				sb.WriteString(wrappingCharacterEnd)
+			}
 		}
 
 		if index < len(pathStack)-1 {
 			sb.WriteString(".")
 		}
 	}
-	return sb.String()
+	var pathString = sb.String()
+	log.Debug("got a path string: %v", pathString)
+	return pathString
 }
 
-func guessKind(head string, tail []string, guess yaml.Kind) yaml.Kind {
+func guessKind(head interface{}, tail []interface{}, guess yaml.Kind) yaml.Kind {
-	log.Debug("tail %v", tail)
+	log.Debug("guessKind: tail %v", tail)
 	if len(tail) == 0 && guess == 0 {
 		log.Debug("end of path, must be a scalar")
 		return yaml.ScalarNode
 	} else if len(tail) == 0 {
 		return guess
 	}
-
+	var next = tail[0]
-	var _, errorParsingInt = strconv.ParseInt(tail[0], 10, 64)
+	switch next.(type) {
-	if tail[0] == "+" || errorParsingInt == nil {
+	case int64:
 		return yaml.SequenceNode
+	default:
+		var nextString = fmt.Sprintf("%v", next)
+		if nextString == "+" {
+			return yaml.SequenceNode
+		}
+		pathParser := NewPathParser()
+		if pathParser.IsPathExpression(nextString) && (guess == yaml.SequenceNode || guess == yaml.MappingNode) {
+			return guess
+		} else if guess == yaml.AliasNode {
+			log.Debug("guess was an alias, okey doke.")
+			return guess
+		} else if head == "**" {
+			log.Debug("deep wildcard, go with the guess")
+			return guess
+		}
+		log.Debug("forcing a mapping node")
+		log.Debug("yaml.SequenceNode %v", guess == yaml.SequenceNode)
+		log.Debug("yaml.ScalarNode %v", guess == yaml.ScalarNode)
+		return yaml.MappingNode
 	}
-	pathParser := NewPathParser()
-	if (pathParser.IsPathExpression(tail[0]) || head == "**") && (guess == yaml.SequenceNode || guess == yaml.MappingNode) {
-		return guess
-	}
-	if guess == yaml.AliasNode {
-		log.Debug("guess was an alias, okey doke.")
-		return guess
-	}
-	log.Debug("forcing a mapping node")
-	log.Debug("yaml.SequenceNode %v", guess == yaml.SequenceNode)
-	log.Debug("yaml.ScalarNode %v", guess == yaml.ScalarNode)
-	return yaml.MappingNode
 }
 
 type YqLib interface {
-	Get(rootNode *yaml.Node, path string) ([]*NodeContext, error)
+	Get(rootNode *yaml.Node, path string, deeplyTraverseArrays bool) ([]*NodeContext, error)
 	Update(rootNode *yaml.Node, updateCommand UpdateCommand, autoCreate bool) error
 	New(path string) yaml.Node
 
@@ -107,9 +151,9 @@ func NewYqLib() YqLib {
 	}
 }
 
-func (l *lib) Get(rootNode *yaml.Node, path string) ([]*NodeContext, error) {
+func (l *lib) Get(rootNode *yaml.Node, path string, deeplyTraverseArrays bool) ([]*NodeContext, error) {
 	var paths = l.parser.ParsePath(path)
-	navigationStrategy := ReadNavigationStrategy()
+	navigationStrategy := ReadNavigationStrategy(deeplyTraverseArrays)
 	navigator := NewDataNavigator(navigationStrategy)
 	error := navigator.Traverse(rootNode, paths)
 	return navigationStrategy.GetVisitedNodes(), error

pkg/yqlib/navigation_strategy.go

@@ -8,13 +8,13 @@ import (
 
 type NodeContext struct {
 	Node      *yaml.Node
-	Head      string
+	Head      interface{}
-	Tail      []string
+	Tail      []interface{}
 	PathStack []interface{}
 }
 
-func NewNodeContext(node *yaml.Node, head string, tail []string, pathStack []interface{}) NodeContext {
+func NewNodeContext(node *yaml.Node, head interface{}, tail []interface{}, pathStack []interface{}) NodeContext {
-	newTail := make([]string, len(tail))
+	newTail := make([]interface{}, len(tail))
 	copy(newTail, tail)
 
 	newPathStack := make([]interface{}, len(pathStack))
@@ -34,18 +34,23 @@ type NavigationStrategy interface {
 	// node key is the string value of the last element in the path stack
 	// we use it to match against the pathExpression in head.
 	ShouldTraverse(nodeContext NodeContext, nodeKey string) bool
+	ShouldDeeplyTraverse(nodeContext NodeContext) bool
+	// when deeply traversing, should we visit all matching nodes, or just leaves?
+	ShouldOnlyDeeplyVisitLeaves(NodeContext) bool
 	GetVisitedNodes() []*NodeContext
 	DebugVisitedNodes()
 	GetPathParser() PathParser
 }
 
 type NavigationStrategyImpl struct {
-	followAlias        func(nodeContext NodeContext) bool
+	followAlias                 func(nodeContext NodeContext) bool
-	autoCreateMap      func(nodeContext NodeContext) bool
+	autoCreateMap               func(nodeContext NodeContext) bool
-	visit              func(nodeContext NodeContext) error
+	visit                       func(nodeContext NodeContext) error
-	shouldVisitExtraFn func(nodeContext NodeContext) bool
+	shouldVisitExtraFn          func(nodeContext NodeContext) bool
-	visitedNodes       []*NodeContext
+	shouldDeeplyTraverse        func(nodeContext NodeContext) bool
-	pathParser         PathParser
+	shouldOnlyDeeplyVisitLeaves func(nodeContext NodeContext) bool
+	visitedNodes                []*NodeContext
+	pathParser                  PathParser
 }
 
 func (ns *NavigationStrategyImpl) GetPathParser() PathParser {
@@ -57,11 +62,32 @@ func (ns *NavigationStrategyImpl) GetVisitedNodes() []*NodeContext {
 }
 
 func (ns *NavigationStrategyImpl) FollowAlias(nodeContext NodeContext) bool {
-	return ns.followAlias(nodeContext)
+	if ns.followAlias != nil {
+		return ns.followAlias(nodeContext)
+	}
+	return true
 }
 
 func (ns *NavigationStrategyImpl) AutoCreateMap(nodeContext NodeContext) bool {
-	return ns.autoCreateMap(nodeContext)
+	if ns.autoCreateMap != nil {
+		return ns.autoCreateMap(nodeContext)
+	}
+	return false
+}
+
+func (ns *NavigationStrategyImpl) ShouldDeeplyTraverse(nodeContext NodeContext) bool {
+	if ns.shouldDeeplyTraverse != nil {
+		return ns.shouldDeeplyTraverse(nodeContext)
+	}
+	return true
+}
+
+func (ns *NavigationStrategyImpl) ShouldOnlyDeeplyVisitLeaves(nodeContext NodeContext) bool {
+	if ns.shouldOnlyDeeplyVisitLeaves != nil {
+		return ns.shouldOnlyDeeplyVisitLeaves(nodeContext)
+	}
+	return true
+
 }
 
 func (ns *NavigationStrategyImpl) ShouldTraverse(nodeContext NodeContext, nodeKey string) bool {
@@ -84,7 +110,6 @@ func (ns *NavigationStrategyImpl) shouldVisit(nodeContext NodeContext) bool {
 		return true
 	}
 	log.Debug("tail len %v", len(nodeContext.Tail))
-	// SOMETHING HERE!
 
 	if ns.alreadyVisited(pathStack) || len(nodeContext.Tail) != 0 {
 		return false

pkg/yqlib/path_parser.go

@@ -1,12 +1,15 @@
 package yqlib
 
 import (
+	"fmt"
 	"strconv"
 	"strings"
+
+	yaml "gopkg.in/yaml.v3"
 )
 
 type PathParser interface {
-	ParsePath(path string) []string
+	ParsePath(path string) []interface{}
 	MatchesNextPathElement(nodeContext NodeContext, nodeKey string) bool
 	IsPathExpression(pathElement string) bool
 }
@@ -42,9 +45,11 @@ func (p *pathParser) MatchesNextPathElement(nodeContext NodeContext, nodeKey str
 	if head == "**" || head == "*" {
 		return true
 	}
-	if strings.Contains(head, "==") {
+	var headString = fmt.Sprintf("%v", head)
+
+	if strings.Contains(headString, "==") && nodeContext.Node.Kind != yaml.ScalarNode {
 		log.Debug("ooh deep recursion time")
-		result := strings.SplitN(head, "==", 2)
+		result := strings.SplitN(headString, "==", 2)
 		path := strings.TrimSpace(result[0])
 		value := strings.TrimSpace(result[1])
 		log.Debug("path %v", path)
@@ -60,6 +65,14 @@ func (p *pathParser) MatchesNextPathElement(nodeContext NodeContext, nodeKey str
 		}
 		log.Debug("done deep recursing, found %v matches", len(navigationStrategy.GetVisitedNodes()))
 		return len(navigationStrategy.GetVisitedNodes()) > 0
+	} else if strings.Contains(headString, "==") && nodeContext.Node.Kind == yaml.ScalarNode {
+		result := strings.SplitN(headString, "==", 2)
+		path := strings.TrimSpace(result[0])
+		value := strings.TrimSpace(result[1])
+		if path == "." {
+			log.Debug("need to match scalar")
+			return matchesString(value, nodeContext.Node.Value)
+		}
 	}
 
 	if head == "+" {
@@ -70,17 +83,18 @@ func (p *pathParser) MatchesNextPathElement(nodeContext NodeContext, nodeKey str
 		}
 	}
 
-	return matchesString(head, nodeKey)
+	return matchesString(headString, nodeKey)
 }
 
-func (p *pathParser) ParsePath(path string) []string {
+func (p *pathParser) ParsePath(path string) []interface{} {
+	var paths = make([]interface{}, 0)
 	if path == "" {
-		return []string{}
+		return paths
 	}
-	return p.parsePathAccum([]string{}, path)
+	return p.parsePathAccum(paths, path)
 }
 
-func (p *pathParser) parsePathAccum(paths []string, remaining string) []string {
+func (p *pathParser) parsePathAccum(paths []interface{}, remaining string) []interface{} {
 	head, tail := p.nextYamlPath(remaining)
 	if tail == "" {
 		return append(paths, head)
@@ -88,11 +102,16 @@ func (p *pathParser) parsePathAccum(paths []string, remaining string) []string {
 	return p.parsePathAccum(append(paths, head), tail)
 }
 
-func (p *pathParser) nextYamlPath(path string) (pathElement string, remaining string) {
+func (p *pathParser) nextYamlPath(path string) (pathElement interface{}, remaining string) {
 	switch path[0] {
 	case '[':
 		// e.g [0].blah.cat -> we need to return "0" and "blah.cat"
-		return p.search(path[1:], []uint8{']'}, true)
+		var value, remainingBit = p.search(path[1:], []uint8{']'}, true)
+		var number, errParsingInt = strconv.ParseInt(value, 10, 64) // nolint
+		if errParsingInt == nil {
+			return number, remainingBit
+		}
+		return value, remainingBit
 	case '"':
 		// e.g "a.b".blah.cat -> we need to return "a.b" and "blah.cat"
 		return p.search(path[1:], []uint8{'"'}, true)

pkg/yqlib/path_parser_test.go

@@ -10,20 +10,21 @@ var parser = NewPathParser()
 
 var parsePathsTests = []struct {
 	path          string
-	expectedPaths []string
+	expectedPaths []interface{}
 }{
-	{"a.b", []string{"a", "b"}},
+	{"a.b", append(make([]interface{}, 0), "a", "b")},
-	{"a.b.**", []string{"a", "b", "**"}},
+	{"a.b.**", append(make([]interface{}, 0), "a", "b", "**")},
-	{"a.b.*", []string{"a", "b", "*"}},
+	{"a.b.*", append(make([]interface{}, 0), "a", "b", "*")},
-	{"a.b[0]", []string{"a", "b", "0"}},
+	{"a.b[0]", append(make([]interface{}, 0), "a", "b", int64(0))},
-	{"a.b.d[+]", []string{"a", "b", "d", "+"}},
+	{"a.b.0", append(make([]interface{}, 0), "a", "b", "0")},
-	{"a", []string{"a"}},
+	{"a.b.d[+]", append(make([]interface{}, 0), "a", "b", "d", "+")},
-	{"a.b.c", []string{"a", "b", "c"}},
+	{"a", append(make([]interface{}, 0), "a")},
-	{"\"a.b\".c", []string{"a.b", "c"}},
+	{"a.b.c", append(make([]interface{}, 0), "a", "b", "c")},
-	{"a.\"b.c\".d", []string{"a", "b.c", "d"}},
+	{"\"a.b\".c", append(make([]interface{}, 0), "a.b", "c")},
-	{"[1].a.d", []string{"1", "a", "d"}},
+	{"a.\"b.c\".d", append(make([]interface{}, 0), "a", "b.c", "d")},
-	{"a[0].c", []string{"a", "0", "c"}},
+	{"[1].a.d", append(make([]interface{}, 0), int64(1), "a", "d")},
-	{"[0]", []string{"0"}},
+	{"a[0].c", append(make([]interface{}, 0), "a", int64(0), "c")},
+	{"[0]", append(make([]interface{}, 0), int64(0))},
 }
 
 func TestPathParserParsePath(t *testing.T) {

pkg/yqlib/read_navigation_strategy.go

@@ -1,17 +1,24 @@
 package yqlib
 
-func ReadNavigationStrategy() NavigationStrategy {
+func ReadNavigationStrategy(deeplyTraverseArrays bool) NavigationStrategy {
 	return &NavigationStrategyImpl{
 		visitedNodes: []*NodeContext{},
 		pathParser:   NewPathParser(),
-		followAlias: func(nodeContext NodeContext) bool {
-			return true
-		},
-		autoCreateMap: func(nodeContext NodeContext) bool {
-			return false
-		},
 		visit: func(nodeContext NodeContext) error {
 			return nil
 		},
+		shouldDeeplyTraverse: func(nodeContext NodeContext) bool {
+			var isInArray = false
+			if len(nodeContext.PathStack) > 0 {
+				var lastElement = nodeContext.PathStack[len(nodeContext.PathStack)-1]
+				switch lastElement.(type) {
+				case int:
+					isInArray = true
+				default:
+					isInArray = false
+				}
+			}
+			return deeplyTraverseArrays || !isInArray
+		},
 	}
 }

pkg/yqlib/update_navigation_strategy.go

@@ -18,11 +18,15 @@ func UpdateNavigationStrategy(updateCommand UpdateCommand, autoCreate bool) Navi
 				DebugNode(node)
 				log.Debug("with")
 				DebugNode(changesToApply)
-				node.Value = changesToApply.Value
+				if !updateCommand.DontUpdateNodeValue {
+					node.Value = changesToApply.Value
+				}
 				node.Tag = changesToApply.Tag
 				node.Kind = changesToApply.Kind
 				node.Style = changesToApply.Style
 				node.Content = changesToApply.Content
+				node.Anchor = changesToApply.Anchor
+				node.Alias = changesToApply.Alias
 				node.HeadComment = changesToApply.HeadComment
 				node.LineComment = changesToApply.LineComment
 				node.FootComment = changesToApply.FootComment

pkg/yqlib/value_parser.go

@@ -5,7 +5,7 @@ import (
 )
 
 type ValueParser interface {
-	Parse(argument string, customTag string) *yaml.Node
+	Parse(argument string, customTag string, customStyle string, anchorName string, createAlias bool) *yaml.Node
 }
 
 type valueParser struct {
@@ -15,9 +15,32 @@ func NewValueParser() ValueParser {
 	return &valueParser{}
 }
 
-func (v *valueParser) Parse(argument string, customTag string) *yaml.Node {
+func (v *valueParser) Parse(argument string, customTag string, customStyle string, anchorName string, createAlias bool) *yaml.Node {
-	if argument == "[]" {
+	var style yaml.Style
-		return &yaml.Node{Tag: "!!seq", Kind: yaml.SequenceNode}
+	if customStyle == "tagged" {
+		style = yaml.TaggedStyle
+	} else if customStyle == "double" {
+		style = yaml.DoubleQuotedStyle
+	} else if customStyle == "single" {
+		style = yaml.SingleQuotedStyle
+	} else if customStyle == "literal" {
+		style = yaml.LiteralStyle
+	} else if customStyle == "folded" {
+		style = yaml.FoldedStyle
+	} else if customStyle == "flow" {
+		style = yaml.FlowStyle
+	} else if customStyle != "" {
+		log.Error("Unknown style %v, ignoring", customStyle)
 	}
-	return &yaml.Node{Value: argument, Tag: customTag, Kind: yaml.ScalarNode}
+	if argument == "[]" {
+		return &yaml.Node{Tag: "!!seq", Kind: yaml.SequenceNode, Style: style}
+	}
+
+	kind := yaml.ScalarNode
+
+	if createAlias {
+		kind = yaml.AliasNode
+	}
+
+	return &yaml.Node{Value: argument, Tag: customTag, Kind: kind, Style: style, Anchor: anchorName}
 }

pkg/yqlib/value_parser_test.go

@@ -7,6 +7,26 @@ import (
 	yaml "gopkg.in/yaml.v3"
 )
 
+var parseStyleTests = []struct {
+	customStyle   string
+	expectedStyle yaml.Style
+}{
+	{"", 0},
+	{"tagged", yaml.TaggedStyle},
+	{"double", yaml.DoubleQuotedStyle},
+	{"single", yaml.SingleQuotedStyle},
+	{"folded", yaml.FoldedStyle},
+	{"flow", yaml.FlowStyle},
+	{"literal", yaml.LiteralStyle},
+}
+
+func TestValueParserStyleTag(t *testing.T) {
+	for _, tt := range parseStyleTests {
+		actual := NewValueParser().Parse("cat", "", tt.customStyle, "", false)
+		test.AssertResultWithContext(t, tt.expectedStyle, actual.Style, tt.customStyle)
+	}
+}
+
 var parseValueTests = []struct {
 	argument        string
 	customTag       string
@@ -20,7 +40,7 @@ var parseValueTests = []struct {
 
 func TestValueParserParse(t *testing.T) {
 	for _, tt := range parseValueTests {
-		actual := NewValueParser().Parse(tt.argument, tt.customTag)
+		actual := NewValueParser().Parse(tt.argument, tt.customTag, "", "", false)
 		test.AssertResultWithContext(t, tt.argument, actual.Value, tt.testDescription)
 		test.AssertResultWithContext(t, tt.expectedTag, actual.Tag, tt.testDescription)
 		test.AssertResult(t, yaml.ScalarNode, actual.Kind)
@@ -28,7 +48,18 @@ func TestValueParserParse(t *testing.T) {
 }
 
 func TestValueParserParseEmptyArray(t *testing.T) {
-	actual := NewValueParser().Parse("[]", "")
+	actual := NewValueParser().Parse("[]", "", "", "", false)
 	test.AssertResult(t, "!!seq", actual.Tag)
 	test.AssertResult(t, yaml.SequenceNode, actual.Kind)
 }
+
+func TestValueParserParseAlias(t *testing.T) {
+	actual := NewValueParser().Parse("bob", "", "", "", true)
+	test.AssertResult(t, "bob", actual.Value)
+	test.AssertResult(t, yaml.AliasNode, actual.Kind)
+}
+
+func TestValueParserAnchorname(t *testing.T) {
+	actual := NewValueParser().Parse("caterpillar", "", "", "foo", false)
+	test.AssertResult(t, "foo", actual.Anchor)
+}

release_instructions.txt

@@ -33,6 +33,8 @@
   - docker build .  -t mikefarah/yq:latest -t mikefarah/yq:VERSION 
 
 - debian package
+  - ensure you get all vendor dependencies before packaging
+	```go mod vendor```
   - execute 
     ```dch -i```
   - fill debian/changelog with changes from last version
@@ -42,4 +44,4 @@
   - put to PPA
     ```dput ppa:<REPOSITORY> ../yq_<VERSION>_source.changes```
     (current distro repository is ppa:rmescandon/yq. In case that a new version
-    is released, please contact rmescandon@gmail.com to bump debian package)
+    is released, please contact rmescandon@gmail.com to bump debian package)

scripts/devtools.sh

@@ -1,4 +1,4 @@
 #!/bin/sh
-set -e
+set -ex
-wget -O- -nv https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s v1.23.1
+go get golang.org/x/tools/cmd/goimports
-go get golang.org/x/tools/cmd/goimports
+wget -O- -nv https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s v1.24.0

scripts/format.sh

@@ -1,3 +1,4 @@
 #!/bin/bash
 
 find . \( -path ./vendor \) -prune -o -name "*.go" -exec goimports -w {} \;
+go mod tidy

scripts/publish.sh

@@ -32,5 +32,5 @@ upload() {
     done < <(find ./build -mindepth 1 -maxdepth 1 -print0)
 }
 
-# release
+release
 upload

scripts/xcompile.sh

@@ -7,3 +7,6 @@ gox -ldflags "${LDFLAGS}" -output="build/yq_{{.OS}}_{{.Arch}}"
 # include non-default linux builds too
 gox -ldflags "${LDFLAGS}" -os=linux  -output="build/yq_{{.OS}}_{{.Arch}}"
 
+cd build
+rhash -r -a . -P -o checksums
+

snap/snapcraft.yaml

@@ -1,9 +1,10 @@
 name: yq
-version: '3.0.1'
+version: '3.3.2'
 summary: A lightweight and portable command-line YAML processor
 description: |
   The aim of the project is to be the jq or sed of yaml files.
 
+base: core18
 grade: stable # devel|stable. must be 'stable' to release into candidate/stable channels
 confinement: strict
 
@@ -15,8 +16,7 @@ apps:
 parts:
   yq:
     plugin: go
+    go-channel: 1.14/stable
     source: .
+    source-type: git
     go-importpath: github.com/mikefarah/yq
-    after: [go]
-  go:
-    source-tag: go1.11

test.yml

@@ -1,3 +0,0 @@
-b:
-  c: thing
-  d: another thing

upgrading-from-v2.md

@@ -0,0 +1,107 @@
+---
+description: New features and breaking changes
+---
+
+# Upgrading from V2
+
+## New Features
+
+* Keeps yaml comments and formatting, can specify yaml [tags](usage/value-parsing.md#using-the-tag-field-to-override) when updating.
+* Handles anchors!
+* Can print out matching paths and values when splatting, more info [here](commands/read.md#printing-matching-paths).
+* JSON output works for all commands! Yaml files with multiple documents are printed out as one JSON document per line, more info [here](usage/convert.md)
+* Deep splat \(`**`\) to match arbitrary paths and match nodes by their children, more info [here](usage/path-expressions.md)
+
+## Breaking Changes
+
+### Parsing values from the CLI
+
+In V3 users are able to better control how values are treated when updating YAML by using a new `--tag` argument \(see more info [here](usage/value-parsing.md)\). A result of this however, is that quoting values, e.g. "true" will no longer have an effect on how the value is interpreted like it did in V2.
+
+For instance, to get the _string_ "true" into a yaml file:
+
+V2:
+
+```text
+yq n a.path '"true"'
+```
+
+V3
+
+```text
+yq n a.path --tag '!!str' true
+```
+
+### Reading paths that don't exist
+
+In V2 this would return null, V3 does not return anything.
+
+Similarly, reading null yaml values `null`, `~` and , V2 returns null whereas V3 returns the values as is.
+
+This is a result of taking effort not to format values coming in and out of the original YAML.
+
+
+
+### Update scripts file format has changed to be more powerful.
+
+Comments can be added, and delete commands have been introduced.
+
+V2
+
+```text
+b.e[+].name: Mike Farah
+```
+
+V3
+
+```yaml
+- command: update 
+  path: b.e[+].thing
+  value:
+    #great 
+    things: frog # wow!
+- command: delete
+  path: b.d
+```
+
+### Reading and splatting, matching results are printed once per line.
+
+e.g:
+
+```yaml
+parent:
+  childA: 
+    no: matches here
+  childB:
+    there: matches
+    hi: no match
+    there2: also matches
+```
+
+```text
+yq r sample.yaml 'parent.*.there*'
+```
+
+V2
+
+```text
+- null
+- - matches
+  - also matches
+```
+
+V3
+
+```text
+matches
+also matches
+```
+
+### Converting JSON to YAML
+
+As JSON is a subset of YAML, and `yq` now preserves the formatting of the passed in document, you will most likely need to use the `--prettyPrint` flag to format the JSON document as idiomatic YAML. See [Working with JSON](usage/convert.md#json-to-yaml) for more info.
+
+
+
+
+

usage/convert.md

@@ -0,0 +1,94 @@
+# Working with JSON
+
+## Yaml to Json
+
+To convert output to json, use the `--tojson` \(or `-j`\) flag. This is supported by all commands. You can change the json output format by using the [pretty print](output-format.md#pretty-print) or [indent](output-format.md#indent) flags. _Note that due to the implementation of the JSON marshaller in GO, object keys will be sorted on output \(_[_https://golang.org/pkg/encoding/json/\#Marshal_](https://golang.org/pkg/encoding/json/#Marshal)_\)._
+
+Given a sample.yaml file of:
+
+```yaml
+b:
+  c: 2
+```
+
+then
+
+```bash
+yq r -j sample.yaml
+```
+
+will output
+
+```javascript
+{"b":{"c":2}}
+```
+
+To format the json:
+
+```yaml
+yq r --prettyPrint -j sample.yaml
+```
+
+will yield
+
+```yaml
+{
+  "b": {
+    "c": 2
+  }
+}
+```
+
+### Multiple matches
+
+Each matching yaml node will be converted to json and printed out on a separate line. The [prettyPrint](output-format.md#pretty-print) and [indent](output-format.md#indent) flags will still work too. ****
+
+Given a sample.yaml file of:
+
+```yaml
+bob:
+  c: 2
+bab:
+  c: 5
+```
+
+then
+
+```bash
+yq r -j sample.yaml b*
+```
+
+will output
+
+```javascript
+{"c":2}
+{"c":5}
+```
+
+## Json to Yaml
+
+To read in json, just pass in a json file instead of yaml, it will just work - as json is a subset of yaml. However, you will probably want to [pretty print the output](output-format.md#pretty-print) to look more like an idiomatic yaml document.
+
+e.g given a json file
+
+```javascript
+{"a":"Easy! as one two three","b":{"c":2,"d":[3,4]}}
+```
+
+then
+
+```bash
+yq r --prettyPrint sample.json
+```
+
+will output
+
+```yaml
+a: Easy! as one two three
+b:
+  c: 2
+  d:
+  - 3
+  - 4
+```
+

usage/output-format.md

@@ -0,0 +1,161 @@
+---
+description: Flags to control yaml and json output format
+---
+
+# Output format
+
+These flags are available for all `yq` commands. 
+
+## Colorize Output
+
+Use the `--colors/-C`flag to print out yaml with colors. This does not work when outputing in JSON format.
+
+## Pretty Print
+
+Use the `--prettyPrint/-P` flag to enforce a formatting style for yaml documents. This is particularly useful when reading a json file \(which is a subset of yaml\) and wanting to format it in a more conventional yaml format.
+
+Given:
+
+```text
+{
+  "apples": [
+    {
+      "are": "great"
+    }
+  ]
+}
+```
+
+Then:
+
+```text
+yq r --prettyPrint sample.json
+```
+
+Will print out:
+
+```text
+apples:
+- are: great
+```
+
+This works in the same manner for yaml files:
+
+```text
+"apples": [are: great]
+```
+
+will format to:
+
+```text
+apples:
+- are: great
+```
+
+## Indent
+
+Use the indent flag `--indent/-I` to control the number of spaces used for indentation. This also works for JSON output. The default value is 2. 
+
+Note that lists are indented at the same level as the map key at indent level 2, but are more deeply indented at indent level 4 and greater. This is \(currently\) a quirk of the underlying [yaml parser](https://github.com/go-yaml/yaml/tree/v3).
+
+Given:
+
+```text
+apples:
+  collection:
+  - name: Green
+  - name: Blue
+  favourite: Pink Lady
+```
+
+Then:
+
+```text
+yq r -I4 sample.yaml
+```
+
+Will print out:
+
+```text
+apples:
+    collection:
+      - name: Green
+      - name: Blue
+    favourite: Pink Lady
+```
+
+With json, you must also specify the `--prettyPrint/-P` flag
+
+```text
+yq r -j -P -I4 sample.yaml
+```
+
+yields
+
+```text
+{
+    "apples": {
+        "collection": [
+            {
+                "name": "Green"
+            },
+            {
+                "name": "Blue"
+            }
+        ],
+        "favourite": "Pink Lady"
+    }
+}
+```
+
+## Unwrap scalars
+
+By default scalar values are 'unwrapped', that is only their value is printed \(except when outputting as JSON\). To print out the node as-is, with the original formatting an any comments pass in `--unwrapScalar=false`
+
+Given data.yml:
+
+```yaml
+a: "Things" # cool stuff
+```
+
+Then:
+
+`yq r --unwrapScalar=false data.yml a`
+
+Will yield:
+
+```yaml
+"Things" # cool stuff
+```
+
+where as without setting the flag to false you would get:
+
+```yaml
+Things
+```
+
+## Strip comments
+
+Use the `--stripComments` flag to print out the yaml file without any of the original comments.
+
+Given data.yml of:
+
+```yaml
+a:
+  b: # there is where the good stuff is
+    c: hi
+```
+
+Then
+
+```yaml
+yq r data.yml a --stripComments
+```
+
+Will yield:
+
+```yaml
+b:
+  c: hi
+```
+

usage/path-expressions.md

@@ -0,0 +1,251 @@
+---
+description: Path expressions are used to deeply navigate and match particular yaml nodes.
+---
+
+# Path Expressions
+
+_As a general rule, you should wrap paths in quotes to prevent your CLI from processing `*`, `[]` and other special characters._
+
+## Simple expressions
+
+### Maps
+
+`'a.b.c'`
+
+```yaml
+a:
+  b:
+    c: thing # MATCHES
+```
+
+### Arrays
+
+`'a.b[1].c'`
+
+```yaml
+a:
+  b:
+  - c: thing0 
+  - c: thing1 # MATCHES
+  - c: thing2
+```
+
+#### Appending to arrays
+
+\(e.g. when using the write command\)
+
+`'a.b[+].c'`
+
+```yaml
+a:
+  b:
+  - c: thing0
+```
+
+Will add a new entry:
+
+```yaml
+a:
+  b:
+  - c: thing0 
+  - c: thing1 # NEW entry from [+] on B array.
+```
+
+#### Negative Array indexes
+
+Negative array indexes can be used to traverse the array in reverse
+
+`'a.b[-1].c'`
+
+Will access the last element in the `b` array and yield:
+
+```yaml
+thing2
+```
+
+## Splat
+
+### Maps
+
+`'a.*.c'`
+
+```yaml
+a:
+  b1:
+    c: thing # MATCHES
+    d: whatever
+  b2:
+    c: thing # MATCHES
+    f: something irrelevant
+```
+
+#### Prefix splat
+
+`'bob.item*.cats'`
+
+```yaml
+bob:
+  item:
+    cats: bananas # MATCHES
+  something:
+    cats: lemons
+  itemThing:
+    cats: more bananas # MATCHES
+  item2:
+    cats: apples # MATCHES
+  thing:
+    cats: oranges
+```
+
+### Arrays
+
+`'a.b[*].c'`
+
+```yaml
+a:
+  b:
+  - c: thing0 # MATCHES
+    d: what..ever
+  - c: thing1 # MATCHES
+    d: blarh
+  - c: thing2 # MATCHES
+    f: thingamabob
+```
+
+## Deep Splat
+
+`**` will match arbitrary nodes for both maps and arrays:
+
+`'a.**.c'`
+
+```yaml
+a:
+  b1:
+    c: thing1 # MATCHES
+    d: cat cat
+  b2:
+    c: thing2 # MATCHES
+    d: dog dog
+  b3:
+    d:
+    - f:
+        c: thing3 # MATCHES
+        d: beep
+    - f:
+        g:
+          c: thing4 # MATCHES
+          d: boop
+    - d: mooo
+```
+
+## Search by children nodes
+
+You can search children by nodes - note that this will return the _parent_ of the matching expression, in the example below the parent\(s\) will be the matching indices of the 'a' array. We can then navigate down to get 'b.c' of each matching indice.
+
+`'a.(b.d==cat).b.c'`
+
+```yaml
+a:
+  - b:
+      c: thing0
+      d: leopard
+    ba: fast
+  - b:
+      c: thing1 # MATCHES
+      d: cat
+    ba: meowy
+  - b:
+      c: thing2
+      d: caterpillar
+    ba: icky
+  - b:
+      c: thing3 # MATCHES
+      d: cat
+    ba: also meowy
+```
+
+### With prefixes
+
+`'a.(b.d==cat*).c'`
+
+```yaml
+a:
+  - b:
+      c: thing0
+      d: leopard
+    ba: fast
+  - b:
+      c: thing1 # MATCHES
+      d: cat
+    ba: meowy
+  - b:
+      c: thing2 # MATCHES
+      d: caterpillar
+    ba: icky
+  - b:
+      c: thing3 # MATCHES
+      d: cat
+    ba: also meowy
+```
+
+### Matching children values
+
+`'animals(.==cat)'`
+
+```yaml
+animals:
+  - dog
+  - cat # MATCHES  
+  - rat
+```
+
+this also works in maps, and with prefixes
+
+`'animals(.==c*)'`
+
+```yaml
+animals:
+  friendliest: cow # MATCHES
+  favourite: cat # MATCHES
+  smallest: rat
+```
+
+## Special Characters
+
+When your yaml field has special characters that overlap with `yq` path expression characters, you will need to escape them in order for the command to work.
+
+### Keys with dots
+
+When specifying a key that has a dot use key lookup indicator.
+
+```yaml
+b:
+  foo.bar: 7
+```
+
+```bash
+yaml r sample.yaml 'b."foo.bar"'
+```
+
+```bash
+yaml w sample.yaml 'b."foo.bar"' 9
+```
+
+Any valid yaml key can be specified as part of a key lookup.
+
+Note that the path is in single quotes to avoid the double quotes being interpreted by your shell.
+
+### Keys \(and values\) with leading dashes
+
+The flag terminator needs to be used to stop the app from attempting to parse the subsequent arguments as flags, if they start if a dash.
+
+```bash
+yq n -j -- --key --value
+```
+
+Will result in
+
+```text
+--key: --value
+```
+

usage/value-parsing.md

@@ -0,0 +1,288 @@
+---
+description: >-
+  How values are parsed from the CLI to commands that create/update yaml (e.g.
+  new/write).
+---
+
+# Value Parsing
+
+`yq` attempts to parse values intelligently, e.g. when a number is passed it - it will assume it's a number as opposed to a string. `yq` will not alter the representation of what you give. So if you pass '03.0' in, it will assume it's a number and keep the value formatted as it was passed in, that is '03.0'.
+
+The `--tag` flag can be used to override the tag type to force particular tags.
+
+## Default behavior
+
+### Integers
+
+_Given_
+
+```bash
+yq new key 3
+```
+
+results in
+
+```yaml
+key: 3
+```
+
+_Given a formatted number_
+
+```bash
+yq new key 03
+```
+
+results in
+
+```yaml
+key: 03
+```
+
+`yq` keeps the number formatted as it was passed in.
+
+### Float
+
+_Given_
+
+```bash
+yq new key "3.1"
+```
+
+results in
+
+```yaml
+key: 3.1
+```
+
+Note that quoting the number does not make a difference.
+
+_Given a formatted decimal number_
+
+```bash
+yq new key 03.0
+```
+
+results in
+
+```yaml
+key: 03.0
+```
+
+`yq` keeps the number formatted as it was passed in
+
+### Booleans
+
+```bash
+yq new key true
+```
+
+results in
+
+```yaml
+key: true
+```
+
+### Nulls
+
+```bash
+yq new key null
+```
+
+results in
+
+```yaml
+key: null
+```
+
+```bash
+yq new key '~'
+```
+
+results in
+
+```yaml
+key: ~
+```
+
+```bash
+yq new key ''
+```
+
+results in
+
+```yaml
+key:
+```
+
+### Strings
+
+```bash
+yq new key whatever
+```
+
+results in
+
+```yaml
+key: whatever
+```
+
+```bash
+yq new key ' whatever '
+```
+
+results in
+
+```yaml
+key: ' whatever '
+```
+
+## Using the tag flag to cast
+
+Previous versions of yq required double quoting to force values to be strings, this no longer works - instead use the --tag flag.
+
+### Casting booleans
+
+```bash
+yq new --tag '!!str' key true
+```
+
+results in
+
+```yaml
+key: "true"
+```
+
+### Casting nulls
+
+```bash
+yq new --tag '!!str' key null
+```
+
+results in
+
+```yaml
+key: "null"
+```
+
+### Custom types
+
+```bash
+yq new --tag '!!farah' key gold
+```
+
+results in
+
+```yaml
+key: !!farah gold
+```
+
+## The style flag
+
+The `--style` flag can be used to specify the quote or block style of the node value. Valid values are
+
+* single
+* double
+* folded
+* flow
+* literal
+* tagged
+
+For example, given:
+
+```bash
+MULTILINE=$(cat <<END
+    This is line one.
+    This is line two.
+END
+)
+
+SINGLE="only one line"
+```
+
+### Single
+
+```yaml
+yq n --style single things "$MULTILINE"
+```
+
+```yaml
+things: 'This is line one.
+
+  This is line two.'
+```
+
+### Double
+
+```yaml
+things: "This is line one.\nThis is line two."
+```
+
+### Folded:
+
+```yaml
+things: >-
+  This is line one.
+
+  This is line two.
+
+```
+
+#### Folded single line:
+
+```yaml
+things: >-
+  only one line
+```
+
+### Flow:
+
+```yaml
+things: |-
+  This is line one.
+  This is line two.
+
+```
+
+#### Flow single line:
+
+```yaml
+things: only one line
+```
+
+### Literal
+
+```yaml
+things: |-
+  This is line one.
+  This is line two.
+
+```
+
+#### Literal single line
+
+```yaml
+things: |-
+  only one line
+```
+
+### Tagged
+
+Always show the tag, note - you must also pass in `--tag='!!str'`
+
+```yaml
+things: !!str |-
+  This is line one.
+  This is line two.
+
+```
+
+#### Tagged single line
+
+```yaml
+things: !!str only one line
+```
+
+
+
+
+

yq.go

@@ -4,14 +4,11 @@ import (
 	"os"
 
 	command "github.com/mikefarah/yq/v3/cmd"
-	logging "gopkg.in/op/go-logging.v1"
 )
 
 func main() {
 	cmd := command.New()
-	log := logging.MustGetLogger("yq")
 	if err := cmd.Execute(); err != nil {
-		log.Error(err.Error())
 		os.Exit(1)
 	}
 }