Array length and collect

Updated all files to be more Debian compliant
Update release instructions for get mod vendor before releasing

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,45 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**Input Yaml**
+Concise yaml document(s) (as simple as possible to show the bug)
+data1.yml:
+```yaml
+this: should really work
+```
+
+data2.yml:
+```yaml
+but: it strangely didn't
+```
+
+**Command**
+The command you ran:
+```
+yq merge data1.yml data2.yml
+```
+
+**Actual behavior**
+
+```yaml
+cat: meow
+```
+
+**Expected behavior**
+
+```yaml
+this: should really work
+but: it strangely didn't
+```
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,36 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+If we have data1.yml like:
+
+```yaml
+country: Australia
+```
+
+And we run a command:
+
+```bash
+yq predictWeather data1.yml
+```
+
+it could output
+
+```yaml
+temp: 32
+```
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/workflows/go.yml

@@ -0,0 +1,30 @@
+name: Build
+on: [push]
+jobs:
+
+  build:
+    name: Build
+    runs-on: ubuntu-latest
+    steps:
+
+    - name: Set up Go 1.13
+      uses: actions/setup-go@v1
+      with:
+        go-version: 1.13
+      id: go
+
+    - name: Check out code into the Go module directory
+      uses: actions/checkout@v2
+
+    - name: Get dependencies
+      run: |
+        go get -v -t -d ./...
+        if [ -f Gopkg.toml ]; then
+            curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
+            dep ensure
+        fi
+
+    - name: Download deps
+      run: scripts/devtools.sh
+    - name: Build
+      run: make local build

.travis.yml

@@ -1,6 +0,0 @@
-language: go
-go:
-  - 1.13.x
-script:
-  - scripts/devtools.sh
-  - make local build

CODE_OF_CONDUCT.md

@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+ advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+ address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at mikefarah@gmail.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

CONTRIBUTING.md

@@ -0,0 +1,8 @@
+1. Install (golang)[https://golang.org/]
+1. Run  `scripts/devtools.sh` to install the required devtools
+2. Run `make [local] vendor` to install the vendor dependencies
+2. Run `make [local] test` to ensure you can run the existing tests
+3. Write unit tests - (see existing examples). Changes will not be accepted without corresponding unit tests.
+4. Make the code changes.
+5. `make [local] test` to lint code and run tests
+6. Profit! ok no profit, but raise a PR and get kudos :)

README.md

@@ -1,6 +1,6 @@
 # yq
 
-[![Build Status](https://api.travis-ci.com/mikefarah/yq.svg?branch=master)](https://travis-ci.com/mikefarah/yq/)  ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq)
+![Build](https://github.com/mikefarah/yq/workflows/Build/badge.svg)  ![Docker Pulls](https://img.shields.io/docker/pulls/mikefarah/yq.svg) ![Github Releases (by Release)](https://img.shields.io/github/downloads/mikefarah/yq/total.svg) ![Go Report](https://goreportcard.com/badge/github.com/mikefarah/yq)
 
 
 a lightweight and portable command-line YAML processor
@@ -12,15 +12,13 @@ V3 is officially out - if you've been using v2 and want/need to upgrade, checkou
 
 ## Install
 
-### Download the latest binary
+### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)
 
-[Here](https://github.com/mikefarah/yq/releases/latest)
+### MacOS:
-
-### On MacOS:
 ```
 brew install yq
 ```
-### On Ubuntu and other Linux distros supporting `snap` packages:
+### Ubuntu and other Linux distros supporting `snap` packages:
 ```
 snap install yq
 ```
@@ -44,11 +42,14 @@ rm /etc/myfile.tmp
 ```
 
 ### On Ubuntu 16.04 or higher from Debian package:
-```
+```sh
+sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys CC86BB64
 sudo add-apt-repository ppa:rmescandon/yq
 sudo apt update
 sudo apt install yq -y
 ```
+Supported by @rmescandon
+
 ### Go Get:
 ```
 GO111MODULE=on go get github.com/mikefarah/yq/v3
@@ -82,7 +83,7 @@ yq() {
 - [List matching paths of a given path expression](https://mikefarah.gitbook.io/yq/commands/read#path-only)
 - Update a yaml file given a [path expression](https://mikefarah.gitbook.io/yq/commands/write-update#basic) or [script file](https://mikefarah.gitbook.io/yq/commands/write-update#basic)
 - Update creates any missing entries in the path on the fly
-- Deeply compare yaml files
+- Deeply [compare](https://mikefarah.gitbook.io/yq/commands/compare) yaml files
 - Keeps yaml formatting and comments when updating
 - [Validate a yaml file](https://mikefarah.gitbook.io/yq/commands/validate)
 - Create a yaml file given a [deep path and value](https://mikefarah.gitbook.io/yq/commands/create#creating-a-simple-yaml-file) or a [script file](https://mikefarah.gitbook.io/yq/commands/create#creating-using-a-create-script)
@@ -114,21 +115,11 @@ Available Commands:
 
 Flags:
   -h, --help          help for yq
+  -I, --indent int    sets indent level for output (default 2)
   -P, --prettyPrint   pretty print
-  -j, --tojson        output as json
+  -j, --tojson        output as json. By default it prints a json document in one line, use the prettyPrint flag to print a formatted doc.
   -v, --verbose       verbose mode
   -V, --version       Print version information and quit
 
 Use "yq [command] --help" for more information about a command.
 ```
-
-## Contribute
-
-**Note: v3 is currently in progress - for the moment I won't be accepting new feature PRs until v3 is ready :)**
-
-1. `scripts/devtools.sh`
-2. `make [local] vendor`
-3. add unit tests
-4. apply changes to go.mod
-5. `make [local] build`
-7. profit

Upgrade Notes

@@ -1,74 +0,0 @@
-
-Major release! Upgraded underlying yaml parser, re-written majority of yq. This has brought on a number of features that have been in demand for a while (see below).
-
-This is in beta and needs some community feedback and testing :)
-
-# New Features
-  - Keeps yaml comments and formatting, can specify yaml tags when updating. https://github.com/mikefarah/yq/issues/19, https://github.com/mikefarah/yq/issues/169, https://github.com/mikefarah/yq/issues/107, https://github.com/mikefarah/yq/issues/171, https://github.com/mikefarah/yq/issues/245, https://github.com/mikefarah/yq/issues/303,https://github.com/mikefarah/yq/issues/308,https://github.com/mikefarah/yq/issues/314
-  - Handles anchors!  https://github.com/mikefarah/yq/issues/310, https://github.com/mikefarah/yq/issues/178
-  - Can print out matching paths and values when splatting https://github.com/mikefarah/yq/issues/20
-  - JSON output works for all commands! Yaml files with multiple documents are printed out as one JSON document per line. 
-  - Deep splat (**) to match arbitrary paths
-
-
-# Breaking changes
-
-## Update scripts file format has changed to be more powerful. 
-Comments can be added, and delete commands have been introduced.
-
-Before:
-```yaml
-b.e[+].name: Mike Farah
-```
-
-After:
-```yaml
-- command: update 
-  path: b.e[+].thing
-  value:
-    #great 
-    things: frog # wow!
-- command: delete
-  path: b.d
-```
-
-https://github.com/mikefarah/yq/issues/305
-
-## Reading and splatting, matching results are printed once per line.
-  e.g:
-
-```json
-parent:
-  childA: 
-    no: matches here
-  childB:
-    there: matches
-    hi: no match
-    there2: also matches
-```
-
-```bash
-yq r sample.yaml 'parent.*.there*'
-```
-
-old
-```yaml
-- null
-- - matches
-  - also matches
-```
-
-new
-```yaml
-matches
-also matches
-```
-
-and you can print the matching paths:
-
-yq r --printMode pv sample.yaml 'parent.*.there*'
-
-```yaml
-parent.childB.there: matches
-parent.childB.there2: also matches
-```

action.yml

@@ -0,0 +1,11 @@
+name: 'YAML processor'
+description: 'YAML processor for running in Github action'
+inputs:
+  cmd:
+    description: 'The Command which should be run'
+    required: true
+runs:
+  using: 'docker'
+  image: 'github-action/Dockerfile'
+  args:
+    - ${{ inputs.cmd }}

cmd/commands_test.go

@@ -91,7 +91,7 @@ func TestReadCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	test.AssertResult(t, "2", result.Output)
+	test.AssertResult(t, "2\n", result.Output)
 }
 
 func TestCompareCmd(t *testing.T) {
@@ -153,16 +153,16 @@ func TestValidateCmd(t *testing.T) {
 
 func TestReadWithAdvancedFilterCmd(t *testing.T) {
 	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "read -v ../examples/sample.yaml b.e(name==sam).value")
+	result := test.RunCmd(cmd, "read ../examples/sample.yaml b.e(name==sam).value")
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	test.AssertResult(t, "4", result.Output)
+	test.AssertResult(t, "4\n", result.Output)
 }
 
 func TestReadWithAdvancedFilterMapCmd(t *testing.T) {
 	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "read -v ../examples/sample.yaml b.e[name==fr*]")
+	result := test.RunCmd(cmd, "read ../examples/sample.yaml b.e[name==fr*]")
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
@@ -183,11 +183,207 @@ func TestReadWithKeyAndValueCmd(t *testing.T) {
 
 func TestReadArrayCmd(t *testing.T) {
 	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "read -p pv ../examples/sample.yaml b.e.1.name")
+	result := test.RunCmd(cmd, "read -p pv ../examples/sample.yaml b.e[1].name")
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	test.AssertResult(t, "b.e.1.name: sam\n", result.Output)
+	test.AssertResult(t, "b.e.[1].name: sam\n", result.Output)
+}
+
+func TestReadArrayLengthCmd(t *testing.T) {
+	content := `- things
+- whatever
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -l %s", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, "2\n", result.Output)
+}
+
+func TestReadArrayLengthDeepCmd(t *testing.T) {
+	content := `holder: 
+- things
+- whatever
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -l %s holder", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, "2\n", result.Output)
+}
+
+func TestReadArrayLengthDeepMultipleCmd(t *testing.T) {
+	content := `holderA: 
+- things
+- whatever
+skipMe:
+- yep
+holderB: 
+- other things
+- cool
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -l -c %s holder*", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, "2\n", result.Output)
+}
+
+func TestReadCollectCmd(t *testing.T) {
+	content := `holderA: yep
+skipMe: not me
+holderB: me too
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -c %s holder*", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `- yep
+- me too
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestReadCollectArrayCmd(t *testing.T) {
+	content := `- name: fred
+  value: 32
+- name: sam
+  value: 67
+- name: fernie
+  value: 103
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -c %s (name==f*)", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `- name: fred
+  value: 32
+- name: fernie
+  value: 103
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestReadArrayLengthDeepMultipleWithPathCmd(t *testing.T) {
+	content := `holderA: 
+- things
+- whatever
+holderB: 
+- other things
+- cool
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -l %s -ppv holder*", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, "holderA: 2\nholderB: 2\n", result.Output)
+}
+
+func TestReadObjectLengthCmd(t *testing.T) {
+	content := `cat: meow
+dog: bark
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -l %s", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, "2\n", result.Output)
+}
+
+func TestReadObjectLengthDeepCmd(t *testing.T) {
+	content := `holder: 
+  cat: meow
+  dog: bark
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -l %s holder", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, "2\n", result.Output)
+}
+
+func TestReadObjectLengthDeepMultipleCmd(t *testing.T) {
+	content := `holderA: 
+  cat: meow
+  dog: bark
+holderB: 
+  elephant: meow
+  zebra: bark
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -l -c %s holder*", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, "2\n", result.Output)
+}
+
+func TestReadObjectLengthDeepMultipleWithPathsCmd(t *testing.T) {
+	content := `holderA: 
+  cat: meow
+  dog: bark
+holderB: 
+  elephant: meow
+  zebra: bark
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -l -ppv %s holder*", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, "holderA: 2\nholderB: 2\n", result.Output)
+}
+
+func TestReadScalarLengthCmd(t *testing.T) {
+	content := `meow`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -l %s", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, "4\n", result.Output)
 }
 
 func TestReadDeepSplatCmd(t *testing.T) {
@@ -226,7 +422,7 @@ func TestReadWithKeyCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	test.AssertResult(t, "b.c", result.Output)
+	test.AssertResult(t, "b.c\n", result.Output)
 }
 
 func TestReadAnchorsCmd(t *testing.T) {
@@ -235,7 +431,7 @@ func TestReadAnchorsCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	test.AssertResult(t, "1", result.Output)
+	test.AssertResult(t, "1\n", result.Output)
 }
 
 func TestReadAnchorsWithKeyAndValueCmd(t *testing.T) {
@@ -279,7 +475,7 @@ func TestReadMergeAnchorsOriginalCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	test.AssertResult(t, "original", result.Output)
+	test.AssertResult(t, "original\n", result.Output)
 }
 
 func TestReadMergeAnchorsExplodeJsonCmd(t *testing.T) {
@@ -318,7 +514,52 @@ pointer: *value-pointer`
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `the value`
+	expectedOutput := "the value\n"
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestReadMergeAnchorsExplodeSimpleArrayCmd(t *testing.T) {
+	content := `- things`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -X %s", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `- things
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestReadNumberKeyJsonCmd(t *testing.T) {
+	content := `data: {"40433437326": 10.833332}`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -j %s", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `{"data":{"40433437326":10.833332}}
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestReadMergeAnchorsExplodeSimpleArrayJsonCmd(t *testing.T) {
+	content := `- things`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read -j %s", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `["things"]
+`
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
@@ -333,7 +574,7 @@ pointer: *value-pointer`
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	expectedOutput := `the value`
+	expectedOutput := "the value\n"
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
@@ -388,7 +629,7 @@ func TestReadMergeAnchorsOverrideCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	test.AssertResult(t, "ice", result.Output)
+	test.AssertResult(t, "ice\n", result.Output)
 }
 
 func TestReadMergeAnchorsPrefixMatchCmd(t *testing.T) {
@@ -410,7 +651,7 @@ func TestReadMergeAnchorsListOriginalCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	test.AssertResult(t, "original", result.Output)
+	test.AssertResult(t, "original\n", result.Output)
 }
 
 func TestReadMergeAnchorsListOverrideInListCmd(t *testing.T) {
@@ -419,7 +660,7 @@ func TestReadMergeAnchorsListOverrideInListCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	test.AssertResult(t, "coconut", result.Output)
+	test.AssertResult(t, "coconut\n", result.Output)
 }
 
 func TestReadMergeAnchorsListOverrideCmd(t *testing.T) {
@@ -428,7 +669,7 @@ func TestReadMergeAnchorsListOverrideCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	test.AssertResult(t, "newbar", result.Output)
+	test.AssertResult(t, "newbar\n", result.Output)
 }
 
 func TestReadInvalidDocumentIndexCmd(t *testing.T) {
@@ -470,7 +711,7 @@ func TestReadMultiCmd(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	test.AssertResult(t, "here", result.Output)
+	test.AssertResult(t, "here\n", result.Output)
 }
 
 func TestReadMultiWithKeyAndValueCmd(t *testing.T) {
@@ -491,7 +732,8 @@ func TestReadMultiAllCmd(t *testing.T) {
 	test.AssertResult(t,
 		`first document
 second document
-third document`, result.Output)
+third document
+`, result.Output)
 }
 
 func TestReadMultiAllWithKeyAndValueCmd(t *testing.T) {
@@ -513,7 +755,7 @@ func TestReadCmd_ArrayYaml(t *testing.T) {
 	if result.Error != nil {
 		t.Error(result.Error)
 	}
-	test.AssertResult(t, "false", result.Output)
+	test.AssertResult(t, "false\n", result.Output)
 }
 
 func TestReadEmptyContentCmd(t *testing.T) {
@@ -530,6 +772,28 @@ func TestReadEmptyContentCmd(t *testing.T) {
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestReadEmptyNodesPrintPathCmd(t *testing.T) {
+	content := `map: 
+  that: {}
+array: 
+  great: []
+null:
+  indeed: ~`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("read %s -ppv **", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `map.that: {}
+array.great: []
+null.indeed: ~
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestReadEmptyContentWithDefaultValueCmd(t *testing.T) {
 	content := ``
 	filename := test.WriteTempYamlFile(content)
@@ -673,7 +937,8 @@ func TestReadCmd_ArrayYaml_SplatWithKeyCmd(t *testing.T) {
 		t.Error(result.Error)
 	}
 	expectedOutput := `[0]
-[1]`
+[1]
+`
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
@@ -684,7 +949,8 @@ func TestReadCmd_ArrayYaml_SplatKey(t *testing.T) {
 		t.Error(result.Error)
 	}
 	expectedOutput := `false
-true`
+true
+`
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
@@ -765,15 +1031,6 @@ func TestReadCmd_ErrorBadPath(t *testing.T) {
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
-func TestReadCmd_Verbose(t *testing.T) {
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, "read -v ../examples/sample.yaml b.c")
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	test.AssertResult(t, "2", result.Output)
-}
-
 func TestReadToJsonCmd(t *testing.T) {
 	cmd := getRootCommand()
 	result := test.RunCmd(cmd, "read -j ../examples/sample.yaml b")
@@ -892,7 +1149,8 @@ b:
 	}
 
 	expectedOutput := `more things
-more things also`
+more things also
+`
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
@@ -947,7 +1205,8 @@ b:
 	}
 
 	expectedOutput := `b.there.c
-b.there2.c`
+b.there2.c
+`
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
@@ -1111,25 +1370,6 @@ func TestPrefixCmd_ErrorUnreadableFile(t *testing.T) {
 	test.AssertResult(t, expectedOutput, result.Error.Error())
 }
 
-func TestPrefixCmd_Verbose(t *testing.T) {
-	content := `b:
-  c: 3
-`
-	filename := test.WriteTempYamlFile(content)
-	defer test.RemoveTempYamlFile(filename)
-
-	cmd := getRootCommand()
-	result := test.RunCmd(cmd, fmt.Sprintf("prefix %s x", filename))
-	if result.Error != nil {
-		t.Error(result.Error)
-	}
-	expectedOutput := `x:
-  b:
-    c: 3
-`
-	test.AssertResult(t, expectedOutput, result.Output)
-}
-
 func TestPrefixCmd_Inplace(t *testing.T) {
 	content := `b:
   c: 3
@@ -1201,6 +1441,79 @@ func TestWriteCmd(t *testing.T) {
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestWriteEmptyMultiDocCmd(t *testing.T) {
+	content := `# this is empty
+---
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s c 7", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `c: 7
+
+# this is empty
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteSurroundingEmptyMultiDocCmd(t *testing.T) {
+	content := `---
+# empty
+---
+cat: frog
+---
+
+# empty
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s -d1 c 7", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `
+
+# empty
+---
+cat: frog
+c: 7
+---
+
+
+# empty
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
+func TestWriteFromFileCmd(t *testing.T) {
+	content := `b:
+  c: 3
+`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	source := `kittens: are cute # sure are!`
+	fromFilename := test.WriteTempYamlFile(source)
+	defer test.RemoveTempYamlFile(fromFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s b.c -f %s", filename, fromFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `b:
+  c:
+    kittens: are cute # sure are!
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestWriteEmptyCmd(t *testing.T) {
 	content := ``
 	filename := test.WriteTempYamlFile(content)
@@ -1217,6 +1530,26 @@ func TestWriteEmptyCmd(t *testing.T) {
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestWriteAutoCreateCmd(t *testing.T) {
+	content := `applications:
+  - name: app
+    env:`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("write %s applications[0].env.hello world", filename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `applications:
+- name: app
+  env:
+    hello: world
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestWriteCmdScript(t *testing.T) {
 	content := `b:
   c: 3
@@ -1519,7 +1852,7 @@ func TestWriteCmd_SplatMapEmpty(t *testing.T) {
 		t.Error(result.Error)
 	}
 	expectedOutput := `b:
-  c: thing
+  c: {}
   d: another thing
 `
 	test.AssertResult(t, expectedOutput, result.Output)
@@ -1812,6 +2145,33 @@ c:
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestMergeAppendArraysCmd(t *testing.T) {
+	content := `people:
+  - name: Barry
+    age: 21`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `people:
+  - name: Roger
+    age: 44`
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge --append -d* %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	expectedOutput := `people:
+- name: Barry
+  age: 21
+- name: Roger
+  age: 44
+`
+	test.AssertResult(t, expectedOutput, result.Output)
+}
+
 func TestMergeOverwriteAndAppendCmd(t *testing.T) {
 	cmd := getRootCommand()
 	result := test.RunCmd(cmd, "merge --autocreate=false --append --overwrite ../examples/data1.yaml ../examples/data2.yaml")
@@ -1890,6 +2250,27 @@ apples: red
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestMergeSpecialCharacterKeysCmd(t *testing.T) {
+	content := ``
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `key[bracket]: value
+key.bracket: value
+key"value": value
+key'value': value
+`
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, mergeContent, result.Output)
+}
+
 func TestMergeYamlMultiAllOverwriteCmd(t *testing.T) {
 	content := `b:
   c: 3
@@ -1920,6 +2301,25 @@ apples: red
 	test.AssertResult(t, expectedOutput, result.Output)
 }
 
+func TestMergeYamlNullMapCmd(t *testing.T) {
+	content := `b:`
+	filename := test.WriteTempYamlFile(content)
+	defer test.RemoveTempYamlFile(filename)
+
+	mergeContent := `b:
+  thing: a frog
+`
+	mergeFilename := test.WriteTempYamlFile(mergeContent)
+	defer test.RemoveTempYamlFile(mergeFilename)
+
+	cmd := getRootCommand()
+	result := test.RunCmd(cmd, fmt.Sprintf("merge %s %s", filename, mergeFilename))
+	if result.Error != nil {
+		t.Error(result.Error)
+	}
+	test.AssertResult(t, mergeContent, result.Output)
+}
+
 func TestMergeCmd_Error(t *testing.T) {
 	cmd := getRootCommand()
 	result := test.RunCmd(cmd, "merge")

cmd/constant.go

@@ -7,11 +7,15 @@ import (
 
 var customTag = ""
 var printMode = "v"
+var printLength = false
+var collectIntoArray = false
 var writeInplace = false
 var writeScript = ""
+var sourceYamlFile = ""
 var outputToJSON = false
 var prettyPrint = false
 var explodeAnchors = false
+var colorsEnabled = false
 var defaultValue = ""
 var indent = 2
 var overwriteFlag = false

cmd/merge.go

@@ -4,6 +4,7 @@ import (
 	"github.com/mikefarah/yq/v3/pkg/yqlib"
 	errors "github.com/pkg/errors"
 	"github.com/spf13/cobra"
+	yaml "gopkg.in/yaml.v3"
 )
 
 func createMergeCmd() *cobra.Command {
@@ -36,6 +37,16 @@ If append flag is set then existing arrays will be merged with the arrays from e
 	return cmdMerge
 }
 
+/*
+* We don't deeply traverse arrays when appending a merge, instead we want to
+* append the entire array element.
+ */
+func createReadFunctionForMerge() func(*yaml.Node) ([]*yqlib.NodeContext, error) {
+	return func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error) {
+		return lib.Get(dataBucket, "**", !appendFlag)
+	}
+}
+
 func mergeProperties(cmd *cobra.Command, args []string) error {
 	var updateCommands []yqlib.UpdateCommand = make([]yqlib.UpdateCommand, 0)
 
@@ -48,7 +59,7 @@ func mergeProperties(cmd *cobra.Command, args []string) error {
 		var filesToMerge = args[1:]
 
 		for _, fileToMerge := range filesToMerge {
-			matchingNodes, errorProcessingFile := readYamlFile(fileToMerge, "**", false, 0)
+			matchingNodes, errorProcessingFile := doReadYamlFile(fileToMerge, createReadFunctionForMerge(), false, 0)
 			if errorProcessingFile != nil {
 				return errorProcessingFile
 			}

cmd/new.go

@@ -1,8 +1,8 @@
 package cmd
 
 import (
+	"github.com/mikefarah/yq/v3/pkg/yqlib"
 	"github.com/spf13/cobra"
-	yaml "gopkg.in/yaml.v3"
 )
 
 func createNewCmd() *cobra.Command {
@@ -46,9 +46,6 @@ func newProperty(cmd *cobra.Command, args []string) error {
 		}
 	}
 
-	var encoder = yaml.NewEncoder(cmd.OutOrStdout())
+	var encoder = yqlib.NewYamlEncoder(cmd.OutOrStdout(), indent, colorsEnabled)
-	encoder.SetIndent(2)
+	return encoder.Encode(&newNode)
-	errorEncoding := encoder.Encode(&newNode)
-	encoder.Close()
-	return errorEncoding
 }

cmd/read.go

@@ -26,6 +26,8 @@ yq r -- things.yaml '--key-starting-with-dashes.blah'
 	cmdRead.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
 	cmdRead.PersistentFlags().StringVarP(&printMode, "printMode", "p", "v", "print mode (v (values, default), p (paths), pv (path and value pairs)")
 	cmdRead.PersistentFlags().StringVarP(&defaultValue, "defaultValue", "D", "", "default value printed when there are no results")
+	cmdRead.PersistentFlags().BoolVarP(&printLength, "length", "l", false, "print length of results")
+	cmdRead.PersistentFlags().BoolVarP(&collectIntoArray, "collect", "c", false, "collect results into array")
 	cmdRead.PersistentFlags().BoolVarP(&explodeAnchors, "explodeAnchors", "X", false, "explode anchors")
 	return cmdRead
 }
@@ -49,6 +51,7 @@ func readProperty(cmd *cobra.Command, args []string) error {
 	if errorReadingStream != nil {
 		return errorReadingStream
 	}
+	out := cmd.OutOrStdout()
 
-	return printResults(matchingNodes, cmd.OutOrStdout())
+	return printResults(matchingNodes, out)
 }

cmd/root.go

@@ -43,6 +43,7 @@ func New() *cobra.Command {
 	rootCmd.PersistentFlags().BoolVarP(&prettyPrint, "prettyPrint", "P", false, "pretty print")
 	rootCmd.PersistentFlags().IntVarP(&indent, "indent", "I", 2, "sets indent level for output")
 	rootCmd.Flags().BoolVarP(&version, "version", "V", false, "Print version information and quit")
+	rootCmd.PersistentFlags().BoolVarP(&colorsEnabled, "colorsEnabled", "C", false, "enable colors")
 
 	rootCmd.AddCommand(
 		createReadCmd(),

cmd/utils.go

@@ -13,7 +13,19 @@ import (
 	yaml "gopkg.in/yaml.v3"
 )
 
+type readDataFn func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error)
+
+func createReadFunction(path string) func(*yaml.Node) ([]*yqlib.NodeContext, error) {
+	return func(dataBucket *yaml.Node) ([]*yqlib.NodeContext, error) {
+		return lib.Get(dataBucket, path, true)
+	}
+}
+
 func readYamlFile(filename string, path string, updateAll bool, docIndexInt int) ([]*yqlib.NodeContext, error) {
+	return doReadYamlFile(filename, createReadFunction(path), updateAll, docIndexInt)
+}
+
+func doReadYamlFile(filename string, readFn readDataFn, updateAll bool, docIndexInt int) ([]*yqlib.NodeContext, error) {
 	var matchingNodes []*yqlib.NodeContext
 
 	var currentIndex = 0
@@ -29,7 +41,7 @@ func readYamlFile(filename string, path string, updateAll bool, docIndexInt int)
 			}
 
 			var errorParsing error
-			matchingNodes, errorParsing = appendDocument(matchingNodes, dataBucket, path, updateAll, docIndexInt, currentIndex)
+			matchingNodes, errorParsing = appendDocument(matchingNodes, dataBucket, readFn, updateAll, docIndexInt, currentIndex)
 			if errorParsing != nil {
 				return errorParsing
 			}
@@ -51,26 +63,42 @@ func handleEOF(updateAll bool, docIndexInt int, currentIndex int) error {
 	return nil
 }
 
-func appendDocument(originalMatchingNodes []*yqlib.NodeContext, dataBucket yaml.Node, path string, updateAll bool, docIndexInt int, currentIndex int) ([]*yqlib.NodeContext, error) {
+func appendDocument(originalMatchingNodes []*yqlib.NodeContext, dataBucket yaml.Node, readFn readDataFn, updateAll bool, docIndexInt int, currentIndex int) ([]*yqlib.NodeContext, error) {
 	log.Debugf("processing document %v - requested index %v", currentIndex, docIndexInt)
 	yqlib.DebugNode(&dataBucket)
 	if !updateAll && currentIndex != docIndexInt {
 		return originalMatchingNodes, nil
 	}
-	log.Debugf("reading %v in document %v", path, currentIndex)
+	log.Debugf("reading in document %v", currentIndex)
-	matchingNodes, errorParsing := lib.Get(&dataBucket, path)
+	matchingNodes, errorParsing := readFn(&dataBucket)
 	if errorParsing != nil {
 		return nil, errors.Wrapf(errorParsing, "Error reading path in document index %v", currentIndex)
 	}
 	return append(originalMatchingNodes, matchingNodes...), nil
 }
 
-func printValue(node *yaml.Node, writer io.Writer) error {
+func lengthOf(node *yaml.Node) int {
-	if node.Kind == yaml.ScalarNode {
+	kindToCheck := node.Kind
-		_, errorWriting := writer.Write([]byte(node.Value))
+	if node.Kind == yaml.DocumentNode && len(node.Content) == 1 {
-		return errorWriting
+		log.Debugf("length of document node, calculating length of child")
+		kindToCheck = node.Content[0].Kind
 	}
-	return printNode(node, writer)
+	switch kindToCheck {
+	case yaml.ScalarNode:
+		return len(node.Value)
+	case yaml.MappingNode:
+		return len(node.Content) / 2
+	default:
+		return len(node.Content)
+	}
+}
+
+// transforms node before printing, if required
+func transformNode(node *yaml.Node) *yaml.Node {
+	if printLength {
+		return &yaml.Node{Kind: yaml.ScalarNode, Value: fmt.Sprintf("%v", lengthOf(node))}
+	}
+	return node
 }
 
 func printNode(node *yaml.Node, writer io.Writer) error {
@@ -78,7 +106,7 @@ func printNode(node *yaml.Node, writer io.Writer) error {
 	if outputToJSON {
 		encoder = yqlib.NewJsonEncoder(writer, prettyPrint, indent)
 	} else {
-		encoder = yqlib.NewYamlEncoder(writer, indent)
+		encoder = yqlib.NewYamlEncoder(writer, indent, colorsEnabled)
 	}
 	return encoder.Encode(node)
 }
@@ -103,10 +131,10 @@ func writeString(writer io.Writer, txt string) error {
 }
 
 func explode(matchingNodes []*yqlib.NodeContext) error {
-
+	log.Debug("exploding nodes")
 	for _, nodeContext := range matchingNodes {
-		var targetNode = yaml.Node{Kind: yaml.MappingNode}
+		var targetNode = yaml.Node{Kind: nodeContext.Node.Kind}
-		explodedNodes, errorRetrieving := lib.Get(nodeContext.Node, "**")
+		explodedNodes, errorRetrieving := lib.Get(nodeContext.Node, "**", true)
 		if errorRetrieving != nil {
 			return errorRetrieving
 		}
@@ -120,6 +148,7 @@ func explode(matchingNodes []*yqlib.NodeContext) error {
 		}
 		nodeContext.Node = &targetNode
 	}
+	log.Debug("done exploding nodes")
 	return nil
 }
 
@@ -147,40 +176,39 @@ func printResults(matchingNodes []*yqlib.NodeContext, writer io.Writer) error {
 		return nil
 	}
 	var errorWriting error
-	for index, mappedDoc := range matchingNodes {
+
+	var arrayCollection = yaml.Node{Kind: yaml.SequenceNode}
+
+	for _, mappedDoc := range matchingNodes {
 		switch printMode {
 		case "p":
-			errorWriting = writeString(bufferedWriter, lib.PathStackToString(mappedDoc.PathStack))
+			errorWriting = writeString(bufferedWriter, lib.PathStackToString(mappedDoc.PathStack)+"\n")
 			if errorWriting != nil {
 				return errorWriting
 			}
-			if index < len(matchingNodes)-1 {
-				errorWriting = writeString(bufferedWriter, "\n")
-				if errorWriting != nil {
-					return errorWriting
-				}
-			}
 		case "pv", "vp":
 			// put it into a node and print that.
 			var parentNode = yaml.Node{Kind: yaml.MappingNode}
 			parentNode.Content = make([]*yaml.Node, 2)
 			parentNode.Content[0] = &yaml.Node{Kind: yaml.ScalarNode, Value: lib.PathStackToString(mappedDoc.PathStack)}
-			parentNode.Content[1] = mappedDoc.Node
+			parentNode.Content[1] = transformNode(mappedDoc.Node)
-			if err := printValue(&parentNode, bufferedWriter); err != nil {
+			if collectIntoArray {
+				arrayCollection.Content = append(arrayCollection.Content, &parentNode)
+			} else if err := printNode(&parentNode, bufferedWriter); err != nil {
 				return err
 			}
 		default:
-			if err := printValue(mappedDoc.Node, bufferedWriter); err != nil {
+			if collectIntoArray {
+				arrayCollection.Content = append(arrayCollection.Content, mappedDoc.Node)
+			} else if err := printNode(transformNode(mappedDoc.Node), bufferedWriter); err != nil {
 				return err
 			}
-			// Printing our Scalars does not print a new line at the end
-			// we only want to do that if there are more values (so users can easily script extraction of values in the yaml)
-			if index < len(matchingNodes)-1 && mappedDoc.Node.Kind == yaml.ScalarNode {
-				errorWriting = writeString(bufferedWriter, "\n")
-				if errorWriting != nil {
-					return errorWriting
 		}
 	}
+
+	if collectIntoArray {
+		if err := printNode(transformNode(&arrayCollection), bufferedWriter); err != nil {
+			return err
 		}
 	}
 
@@ -200,6 +228,11 @@ func parseDocumentIndex() (bool, int, error) {
 
 type updateDataFn func(dataBucket *yaml.Node, currentIndex int) error
 
+func isNullDocument(dataBucket *yaml.Node) bool {
+	return dataBucket.Kind == yaml.DocumentNode && (len(dataBucket.Content) == 0 ||
+		dataBucket.Content[0].Kind == yaml.ScalarNode && dataBucket.Content[0].Tag == "!!null")
+}
+
 func mapYamlDecoder(updateData updateDataFn, encoder yqlib.Encoder) yamlDecoderFn {
 	return func(decoder *yaml.Decoder) error {
 		var dataBucket yaml.Node
@@ -219,8 +252,11 @@ func mapYamlDecoder(updateData updateDataFn, encoder yqlib.Encoder) yamlDecoderF
 
 			if errorReading == io.EOF && docIndexInt == 0 && currentIndex == 0 {
 				//empty document, lets just make one
-				child := yaml.Node{Kind: yaml.MappingNode}
 				dataBucket = yaml.Node{Kind: yaml.DocumentNode, Content: make([]*yaml.Node, 1)}
+				child := yaml.Node{Kind: yaml.MappingNode}
+				dataBucket.Content[0] = &child
+			} else if isNullDocument(&dataBucket) && (updateAll || docIndexInt == currentIndex) {
+				child := yaml.Node{Kind: yaml.MappingNode}
 				dataBucket.Content[0] = &child
 			} else if errorReading == io.EOF {
 				if !updateAll && currentIndex <= docIndexInt {
@@ -325,8 +361,9 @@ func readAndUpdate(stdOut io.Writer, inputFile string, updateData updateDataFn)
 	if outputToJSON {
 		encoder = yqlib.NewJsonEncoder(bufferedWriter, prettyPrint, indent)
 	} else {
-		encoder = yqlib.NewYamlEncoder(bufferedWriter, indent)
+		encoder = yqlib.NewYamlEncoder(bufferedWriter, indent, colorsEnabled)
 	}
+
 	return readStream(inputFile, mapYamlDecoder(updateData, encoder))
 }
 
@@ -355,6 +392,17 @@ func readUpdateCommands(args []string, expectedArgs int, badArgsMessage string)
 		}
 
 		log.Debugf("Read write commands file '%v'", updateCommands)
+	} else if sourceYamlFile != "" && len(args) == expectedArgs-1 {
+		log.Debugf("Reading value from %v", sourceYamlFile)
+		var value yaml.Node
+		err := readData(sourceYamlFile, 0, &value)
+		if err != nil && err != io.EOF {
+			return nil, err
+		}
+		log.Debug("args %v", args[expectedArgs-2])
+		updateCommands = make([]yqlib.UpdateCommand, 1)
+		updateCommands[0] = yqlib.UpdateCommand{Command: "update", Path: args[expectedArgs-2], Value: value.Content[0], Overwrite: true}
+
 	} else if len(args) < expectedArgs {
 		return nil, errors.New(badArgsMessage)
 	} else {

cmd/version.go

@@ -11,7 +11,7 @@ var (
 	GitDescribe string
 
 	// Version is main version number that is being run at the moment.
-	Version = "3.1.0"
+	Version = "3.1.2"
 
 	// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
 	// then it means that it is a final release. Otherwise, this is a pre-release

cmd/write.go

@@ -43,6 +43,7 @@ format is list of update commands (update or delete) like so:
 	}
 	cmdWrite.PersistentFlags().BoolVarP(&writeInplace, "inplace", "i", false, "update the yaml file inplace")
 	cmdWrite.PersistentFlags().StringVarP(&writeScript, "script", "s", "", "yaml script for updating yaml")
+	cmdWrite.PersistentFlags().StringVarP(&sourceYamlFile, "from", "f", "", "yaml file for updating yaml (as-is)")
 	cmdWrite.PersistentFlags().StringVarP(&customTag, "tag", "t", "", "set yaml tag (e.g. !!int)")
 	cmdWrite.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
 	return cmdWrite

debian/changelog

@@ -1,3 +1,31 @@
+yq (3.1-2) eoan; urgency=medium
+
+  * Bug fix: yq 3 was removing empty inline-style objects and arrays (#355)
+  * Bug fix: Merge option returned different output when switching order of 
+    merging files(#347)
+  * Bug fix: Add new object to existing array object was failing in 3.1.1 (#361)
+  * Bug fix: yq 3 empty keys did not allow merging of values (#356)
+  * Bug fix: keys quoted during merge (#363)
+  * Bug fix: Correct length with wc -l (#362)
+  * Bug fix: Write to empty document removed path (#359)
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Mon, 24 Feb 2020 20:31:58 +0100
+
+yq (3.1-1) eoan; urgency=medium
+
+  * Keeps yaml comments and formatting, can specify yaml tags when updating.
+  * Handles anchors
+  * Can print out matching paths and values when splatting
+  * JSON output works for all commands
+  * Yaml files with multiple documents are printed out as one JSON 
+    document per line.
+  * Deep splat (**) to match arbitrary paths
+  * Update scripts file format has changed to be more powerful
+  * Reading and splatting, matching results are printed once per line
+  * Bugfixing
+
+ -- Roberto Mier Escandon <rmescandon@gmail.com>  Tue, 11 Feb 2020 22:18:24 +0100
+
 yq (2.2-1) bionic; urgency=medium
 
   * Added Windows support for the "--inplace" command flag

debian/compat

@@ -1 +1 @@
-9
+10

debian/control

@@ -1,22 +1,22 @@
 Source: yq
 Section: devel
-Priority: extra
+Priority: optional
 Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
-Build-Depends: debhelper (>= 9),
+Build-Depends: debhelper (>=10),
-               dh-golang,
+               dh-golang (>=1.34),
-               golang-1.10-go,
+               golang-1.13,
                rsync
-Standards-Version: 3.9.6
+Standards-Version: 4.1.4
 Homepage: https://github.com/mikefarah/yq.git
 Vcs-Browser: https://github.com/mikefarah/yq.git
 Vcs-Git: https://github.com/mikefarah/yq.git
+XS-Go-Import-Path: github.com/mikefarah/yq
+XSBC-Original-Maintainer: Roberto Mier Escandón <rmescandon@gmail.com>
 
 Package: yq
 Architecture: any
-Built-Using: ${misc:Built-Using}
+Depends: ${shlibs:Depends}, ${misc:Depends}
-Depends: ${shlibs:Depends},
+Description: lightweight and portable command-line YAML processor
-         ${misc:Depends}
-Description:
- a lightweight and portable command-line YAML processor
  .
- The aim of the project is to be the [jq](https://github.com/stedolan/jq) or sed of yaml files.
+ The aim of the project is to be the 
+ [jq](https://github.com/stedolan/jq) or sed of yaml files.

debian/copyright

@@ -3,5 +3,22 @@ Upstream-Name: yq
 Source: https://github.com/mikefarah/yq.git
 
 Files: *
-Copyright: 2017 Mike Farah Ltd. All rights reserved
+Copyright: 2017 Mike Farah
-License: Proprietary
+License: Expat
+    Permission is hereby granted, free of charge, to any person obtaining a copy
+    of this software and associated documentation files (the "Software"), to deal
+    in the Software without restriction, including without limitation the rights
+    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+    copies of the Software, and to permit persons to whom the Software is
+    furnished to do so, subject to the following conditions:
+    .
+    The above copyright notice and this permission notice shall be included in all
+    copies or substantial portions of the Software.
+    .
+    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+    SOFTWARE.

debian/files

@@ -0,0 +1 @@
+yq_3.1-2_source.buildinfo devel optional

debian/rules

@@ -14,46 +14,44 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-
 PROJECT := yq
 OWNER := mikefarah
 REPO := github.com
-GOVERSION := 1.10
+GOVERSION := 1.13
+
 export DH_OPTIONS
 export DH_GOPKG := ${REPO}/${OWNER}/${PROJECT}
 export GOROOT := /usr/lib/go-${GOVERSION}
 export GOPATH := ${CURDIR}/_build
 export GOBIN := ${GOPATH}/bin
 export PATH := ${GOROOT}/bin:${GOBIN}:${PATH}
-BLDPATH := $(shell dpkg-architecture -qDEB_BUILD_GNU_TYPE)
+export GOCACHE := /tmp/gocache
-SRCDIR := ${CURDIR}/_build/src/${DH_GOPKG}
+export GOFLAGS := -mod=vendor
+
+SRCDIR := ${GOPATH}/src/${DH_GOPKG}
 DESTDIR := ${CURDIR}/debian/${PROJECT}
 BINDIR := /usr/bin
 ASSETSDIR := /usr/share/${PROJECT}
 
 %:
-	dh $@ --buildsystem=golang --with=golang
+	dh $@ --builddirectory=${GOPATH} --buildsystem=golang
 
 override_dh_auto_build:
 	mkdir -p ${SRCDIR}
 	mkdir -p ${GOBIN}
 	# copy project to local srcdir to build from there
-	rsync -avz --progress --exclude=obj-${BLDPATH} --exclude=debian . $(SRCDIR)
+	rsync -avz --progress --exclude=_build --exclude=debian --exclude=tmp. --exclude=go.mod --exclude=docs . $(SRCDIR)
 	# build go code
-	(cd ${SRCDIR} && go install ./...)
+	(cd ${SRCDIR} && go install -buildmode=pie ./...)
 	
 override_dh_auto_test:
 	(cd ${SRCDIR} && go test -v ./...)
 
 override_dh_auto_install:
-	mkdir -p ${DESTDIR}/${BINDIR}
+	cp ${GOBIN}/yq ${DESTDIR}/${BINDIR}
-	mkdir -p ${DESTDIR}/${ASSETSDIR}
+	cp -f ${SRCDIR}/LICENSE ${DESTDIR}/${ASSETSDIR}
-	cp ${CURDIR}/_build/bin/yq ${DESTDIR}/${BINDIR}
-	cp -rf ${SRCDIR}/LICENSE ${DESTDIR}/${ASSETSDIR}
-	cp -rf ${SRCDIR}/README.md ${DESTDIR}/${PLUGINSDIR}
 	chmod a+x ${DESTDIR}/${BINDIR}/yq
 
 override_dh_auto_clean:
 	dh_clean
-	rm -rf ${CURDIR}/obj-${BLDPATH}
 	rm -rf ${CURDIR}/_build

debian/yq.dirs

@@ -0,0 +1,3 @@
+usr/bin
+usr/share/yq
+usr/share/man/man1

github-action/Dockerfile

@@ -0,0 +1,5 @@
+FROM mikefarah/yq:3
+
+COPY entrypoint.sh /entrypoint.sh
+
+ENTRYPOINT ["/entrypoint.sh"]

github-action/entrypoint.sh

@@ -0,0 +1,4 @@
+#!/bin/sh -l
+
+echo "$1"
+eval $1

go.mod

@@ -1,13 +1,16 @@
 module github.com/mikefarah/yq/v3
 
 require (
+	github.com/fatih/color v1.9.0
+	github.com/goccy/go-yaml v1.3.2
+	github.com/kr/pretty v0.1.0 // indirect
 	github.com/kylelemons/godebug v1.1.0
 	github.com/pkg/errors v0.8.1
 	github.com/spf13/cobra v0.0.5
-	golang.org/x/tools v0.0.0-20191213221258-04c2e8eff935 // indirect
+	github.com/spf13/pflag v1.0.5 // indirect
-	gopkg.in/imdario/mergo.v0 v0.3.7 // indirect
+	gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 // indirect
 	gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473
-	gopkg.in/yaml.v3 v3.0.0-20191120175047-4206685974f2
+	gopkg.in/yaml.v3 v3.0.0-20200121175148-a6ecf24a6d71
 )
 
 go 1.13

go.sum

@@ -4,19 +4,34 @@ github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc
 github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
 github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
 github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/fatih/color v1.7.0/go.mod h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4=
+github.com/fatih/color v1.9.0 h1:8xPHl4/q1VyqGIPif1F+1V3Y3lSmrq01EabUW3CoW5s=
+github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL+zU=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
+github.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=
+github.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=
+github.com/goccy/go-yaml v1.3.2 h1:joykVKVARE+kQNoaj0ijjPY7lhgdovyU6etuYEl3hFU=
+github.com/goccy/go-yaml v1.3.2/go.mod h1:PsEEJ29nIFZL07P/c8dv4P6rQkVFFXafQee85U+ERHA=
 github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
 github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
+github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
 github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
+github.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=
 github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
-github.com/mikefarah/yaml v2.1.0+incompatible h1:nu2cqmzk4WlWJNgnevY88faMcdrDzYGcsUjYFxEpB7Y=
+github.com/mattn/go-colorable v0.1.4 h1:snbPLB8fVfU9iwbbo30TPtbLRzwWu6aJS6Xh4eaaviA=
-github.com/mikefarah/yaml/v2 v2.4.0 h1:eYqfooY0BnvKTJxr7+ABJs13n3dg9n347GScDaU2Lww=
+github.com/mattn/go-colorable v0.1.4/go.mod h1:U0ppj6V5qS13XJ6of8GYAs25YV2eR4EVcfRqFIhoBtE=
-github.com/mikefarah/yaml/v2 v2.4.0/go.mod h1:ahVqZF4n1W4NqwvVnZzC4es67xsW9uR/RRf2RRxieJU=
+github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
-github.com/mikefarah/yq v2.4.0+incompatible h1:oBxbWy8R9hI3BIUUxEf0CzikWa2AgnGrGhvGQt5jgjk=
+github.com/mattn/go-isatty v0.0.10/go.mod h1:qgIWMr58cqv1PHHyhnkY9lrL7etaEgOFcMEpPG5Rm84=
-github.com/mikefarah/yq/v2 v2.4.1 h1:tajDonaFK6WqitSZExB6fKlWQy/yCkptqxh2AXEe3N4=
+github.com/mattn/go-isatty v0.0.11 h1:FxPOTFNqGkuDUGi3H/qkUbQO4ZiBa2brKq5r0l8TGeM=
+github.com/mattn/go-isatty v0.0.11/go.mod h1:PhnuNfih5lzO57/f3n+odYbM4JtupLOxQOAqxQCu2WE=
 github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
 github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
 github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
@@ -31,34 +46,36 @@ github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tL
 github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
 github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
 github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
+github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
+github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
 github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
 github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
 golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
-golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
-golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
-golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
-golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
-golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190222072716-a9d3bda3a223/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191008105621-543471e840be/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191010194322-b09406accb47/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191026070338-33540a1f6037 h1:YyJpGZS1sBuBCzLAR1VEpK193GlqGZbnPFnPV/5Rsb4=
+golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
-golang.org/x/tools v0.0.0-20191030203535-5e247c9ad0a0 h1:s5lp4ug7qHzUccgyFdjsX7OZDzHXRaePrF3B3vmUiuM=
+golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
-golang.org/x/tools v0.0.0-20191030203535-5e247c9ad0a0/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
-golang.org/x/tools v0.0.0-20191213221258-04c2e8eff935 h1:kJQZhwFzSwJS2BxboKjdZzWczQOZx8VuH7Y8hhuGUtM=
+golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898 h1:/atklqdjdhuosWIl6AIbOeHJjicWYPqR9bpxqxYG2pA=
-golang.org/x/tools v0.0.0-20191213221258-04c2e8eff935/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
-golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
-gopkg.in/imdario/mergo.v0 v0.3.7 h1:QDotlIZtaO/p+Um0ok18HRTpq5i5/SAk/qprsor+9c8=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
-gopkg.in/imdario/mergo.v0 v0.3.7/go.mod h1:9qPP6AGrlC1G2PTNXko614FwGZvorN7MiBU0Eppok+U=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/go-playground/assert.v1 v1.2.1/go.mod h1:9RXL0bg/zibRAgZUYszZSwO/z8Y/a8bDuhia5mkpMnE=
+gopkg.in/go-playground/validator.v9 v9.30.0/go.mod h1:+c9/zcJMFNgbLvly1L1V+PpxWdVbfP1avr/N00E2vyQ=
 gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473 h1:6D+BvnJ/j6e222UW8s2qTSe3wGBtvo0MbVQG/c5k8RE=
 gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473/go.mod h1:N1eN2tsCx0Ydtgjl4cqmbRCsY4/+z4cYDeqwZTk6zog=
 gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v3 v3.0.0-20191120175047-4206685974f2 h1:XZx7nhd5GMaZpmDaEHFVafUZC7ya0fuo7cSJ3UCKYmM=
 gopkg.in/yaml.v3 v3.0.0-20191120175047-4206685974f2/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+gopkg.in/yaml.v3 v3.0.0-20200121175148-a6ecf24a6d71 h1:Xe2gvTZUJpsvOWUnvmL/tmhVBZUmHSvLbMjRj6NUUKo=
+gopkg.in/yaml.v3 v3.0.0-20200121175148-a6ecf24a6d71/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

pkg/yqlib/color_print.go

@@ -0,0 +1,61 @@
+package yqlib
+
+import (
+	"fmt"
+	"io"
+
+	"github.com/fatih/color"
+	"github.com/goccy/go-yaml/lexer"
+	"github.com/goccy/go-yaml/printer"
+)
+
+// Thanks @risentveber!
+
+const escape = "\x1b"
+
+func format(attr color.Attribute) string {
+	return fmt.Sprintf("%s[%dm", escape, attr)
+}
+
+func ColorizeAndPrint(bytes []byte, writer io.Writer) error {
+	tokens := lexer.Tokenize(string(bytes))
+	var p printer.Printer
+	p.Bool = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgHiMagenta),
+			Suffix: format(color.Reset),
+		}
+	}
+	p.Number = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgHiMagenta),
+			Suffix: format(color.Reset),
+		}
+	}
+	p.MapKey = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgCyan),
+			Suffix: format(color.Reset),
+		}
+	}
+	p.Anchor = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgHiYellow),
+			Suffix: format(color.Reset),
+		}
+	}
+	p.Alias = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgHiYellow),
+			Suffix: format(color.Reset),
+		}
+	}
+	p.String = func() *printer.Property {
+		return &printer.Property{
+			Prefix: format(color.FgGreen),
+			Suffix: format(color.Reset),
+		}
+	}
+	_, err := writer.Write([]byte(p.PrintTokens(tokens) + "\n"))
+	return err
+}

pkg/yqlib/data_navigator.go

@@ -1,13 +1,14 @@
 package yqlib
 
 import (
+	"fmt"
 	"strconv"
 
 	yaml "gopkg.in/yaml.v3"
 )
 
 type DataNavigator interface {
-	Traverse(value *yaml.Node, path []string) error
+	Traverse(value *yaml.Node, path []interface{}) error
 }
 
 type navigator struct {
@@ -20,7 +21,7 @@ func NewDataNavigator(NavigationStrategy NavigationStrategy) DataNavigator {
 	}
 }
 
-func (n *navigator) Traverse(value *yaml.Node, path []string) error {
+func (n *navigator) Traverse(value *yaml.Node, path []interface{}) error {
 	realValue := value
 	emptyArray := make([]interface{}, 0)
 	if realValue.Kind == yaml.DocumentNode {
@@ -30,17 +31,14 @@ func (n *navigator) Traverse(value *yaml.Node, path []string) error {
 	return n.doTraverse(value, "", path, emptyArray)
 }
 
-func (n *navigator) doTraverse(value *yaml.Node, head string, tail []string, pathStack []interface{}) error {
+func (n *navigator) doTraverse(value *yaml.Node, head interface{}, tail []interface{}, pathStack []interface{}) error {
-
-	if value.Kind == yaml.ScalarNode {
-		return n.navigationStrategy.Visit(NewNodeContext(value, head, tail, pathStack))
-	}
 
 	log.Debug("head %v", head)
 	DebugNode(value)
+	var nodeContext = NewNodeContext(value, head, tail, pathStack)
 
 	var errorDeepSplatting error
-	if head == "**" {
+	if head == "**" && value.Kind != yaml.ScalarNode && n.navigationStrategy.ShouldDeeplyTraverse(nodeContext) {
 		if len(pathStack) == 0 || pathStack[len(pathStack)-1] != "<<" {
 			errorDeepSplatting = n.recurse(value, head, tail, pathStack)
 		}
@@ -52,59 +50,59 @@ func (n *navigator) doTraverse(value *yaml.Node, head string, tail []string, pat
 		return errorDeepSplatting
 	}
 
-	if len(tail) > 0 {
+	if len(tail) > 0 && value.Kind != yaml.ScalarNode {
 		log.Debugf("diving into %v", tail[0])
 		DebugNode(value)
 		return n.recurse(value, tail[0], tail[1:], pathStack)
 	}
-	return n.navigationStrategy.Visit(NewNodeContext(value, head, tail, pathStack))
+	return n.navigationStrategy.Visit(nodeContext)
 }
 
 func (n *navigator) getOrReplace(original *yaml.Node, expectedKind yaml.Kind) *yaml.Node {
 	if original.Kind != expectedKind {
-		log.Debug("wanted %v but it was %v, overriding", expectedKind, original.Kind)
+		log.Debug("wanted %v but it was %v, overriding", KindString(expectedKind), KindString(original.Kind))
 		return &yaml.Node{Kind: expectedKind}
 	}
 	return original
 }
 
-func (n *navigator) recurse(value *yaml.Node, head string, tail []string, pathStack []interface{}) error {
+func (n *navigator) recurse(value *yaml.Node, head interface{}, tail []interface{}, pathStack []interface{}) error {
 	log.Debug("recursing, processing %v, pathStack %v", head, pathStackToString(pathStack))
 	switch value.Kind {
 	case yaml.MappingNode:
 		log.Debug("its a map with %v entries", len(value.Content)/2)
-		return n.recurseMap(value, head, tail, pathStack)
+		headString := fmt.Sprintf("%v", head)
+		return n.recurseMap(value, headString, tail, pathStack)
+
 	case yaml.SequenceNode:
 		log.Debug("its a sequence of %v things!", len(value.Content))
 
-		var index, errorParsingIndex = strconv.ParseInt(head, 10, 64) // nolint
+		switch head := head.(type) {
-		if errorParsingIndex == nil {
+		case int64:
-			return n.recurseArray(value, index, head, tail, pathStack)
+			return n.recurseArray(value, head, head, tail, pathStack)
-		} else if head == "+" {
+		default:
+
+			if head == "+" {
 				return n.appendArray(value, head, tail, pathStack)
+			} else if len(value.Content) == 0 && head == "**" {
+				return n.navigationStrategy.Visit(NewNodeContext(value, head, tail, pathStack))
 			}
 			return n.splatArray(value, head, tail, pathStack)
-
+		}
 	case yaml.AliasNode:
 		log.Debug("its an alias!")
 		DebugNode(value.Alias)
 		if n.navigationStrategy.FollowAlias(NewNodeContext(value, head, tail, pathStack)) {
-
-			if value.Alias.Kind == yaml.ScalarNode {
-				log.Debug("alias to a scalar")
-				return n.navigationStrategy.Visit(NewNodeContext(value.Alias, head, tail, pathStack))
-			} else {
 			log.Debug("following the alias")
 			return n.recurse(value.Alias, head, tail, pathStack)
 		}
-		}
 		return nil
 	default:
-		return nil
+		return n.navigationStrategy.Visit(NewNodeContext(value, head, tail, pathStack))
 	}
 }
 
-func (n *navigator) recurseMap(value *yaml.Node, head string, tail []string, pathStack []interface{}) error {
+func (n *navigator) recurseMap(value *yaml.Node, head string, tail []interface{}, pathStack []interface{}) error {
 	traversedEntry := false
 	errorVisiting := n.visitMatchingEntries(value, head, tail, pathStack, func(contents []*yaml.Node, indexInMap int) error {
 		log.Debug("recurseMap: visitMatchingEntries for %v", contents[indexInMap].Value)
@@ -115,7 +113,7 @@ func (n *navigator) recurseMap(value *yaml.Node, head string, tail []string, pat
 		if n.navigationStrategy.ShouldTraverse(NewNodeContext(contents[indexInMap+1], head, tail, newPathStack), contents[indexInMap].Value) {
 			log.Debug("recurseMap: Going to traverse")
 			traversedEntry = true
-			// contents[indexInMap+1] = n.getOrReplace(contents[indexInMap+1], guessKind(head, tail, contents[indexInMap+1].Kind))
+			contents[indexInMap+1] = n.getOrReplace(contents[indexInMap+1], guessKind(head, tail, contents[indexInMap+1].Kind))
 			errorTraversing := n.doTraverse(contents[indexInMap+1], head, tail, newPathStack)
 			log.Debug("recurseMap: Finished traversing")
 			n.navigationStrategy.DebugVisitedNodes()
@@ -130,22 +128,33 @@ func (n *navigator) recurseMap(value *yaml.Node, head string, tail []string, pat
 		return errorVisiting
 	}
 
-	if traversedEntry || n.navigationStrategy.GetPathParser().IsPathExpression(head) || !n.navigationStrategy.AutoCreateMap(NewNodeContext(value, head, tail, pathStack)) {
+	if len(value.Content) == 0 && head == "**" {
+		return n.navigationStrategy.Visit(NewNodeContext(value, head, tail, pathStack))
+	} else if traversedEntry || n.navigationStrategy.GetPathParser().IsPathExpression(head) || !n.navigationStrategy.AutoCreateMap(NewNodeContext(value, head, tail, pathStack)) {
 		return nil
 	}
 
+	_, errorParsingInt := strconv.ParseInt(head, 10, 64)
+
 	mapEntryKey := yaml.Node{Value: head, Kind: yaml.ScalarNode}
+
+	if errorParsingInt == nil {
+		// fixes a json encoding problem where keys that look like numbers
+		// get treated as numbers and cannot be used in a json map
+		mapEntryKey.Style = yaml.LiteralStyle
+	}
+
 	value.Content = append(value.Content, &mapEntryKey)
 	mapEntryValue := yaml.Node{Kind: guessKind(head, tail, 0)}
 	value.Content = append(value.Content, &mapEntryValue)
-	log.Debug("adding new node %v", head)
+	log.Debug("adding a new node %v - def a string", head)
 	return n.doTraverse(&mapEntryValue, head, tail, append(pathStack, head))
 }
 
 // need to pass the node in, as it may be aliased
 type mapVisitorFn func(contents []*yaml.Node, index int) error
 
-func (n *navigator) visitDirectMatchingEntries(node *yaml.Node, head string, tail []string, pathStack []interface{}, visit mapVisitorFn) error {
+func (n *navigator) visitDirectMatchingEntries(node *yaml.Node, head string, tail []interface{}, pathStack []interface{}, visit mapVisitorFn) error {
 	var contents = node.Content
 	for index := 0; index < len(contents); index = index + 2 {
 		content := contents[index]
@@ -160,7 +169,7 @@ func (n *navigator) visitDirectMatchingEntries(node *yaml.Node, head string, tai
 	return nil
 }
 
-func (n *navigator) visitMatchingEntries(node *yaml.Node, head string, tail []string, pathStack []interface{}, visit mapVisitorFn) error {
+func (n *navigator) visitMatchingEntries(node *yaml.Node, head string, tail []interface{}, pathStack []interface{}, visit mapVisitorFn) error {
 	var contents = node.Content
 	log.Debug("visitMatchingEntries %v", head)
 	DebugNode(node)
@@ -176,7 +185,7 @@ func (n *navigator) visitMatchingEntries(node *yaml.Node, head string, tail []st
 	return n.visitAliases(contents, head, tail, pathStack, visit)
 }
 
-func (n *navigator) visitAliases(contents []*yaml.Node, head string, tail []string, pathStack []interface{}, visit mapVisitorFn) error {
+func (n *navigator) visitAliases(contents []*yaml.Node, head string, tail []interface{}, pathStack []interface{}, visit mapVisitorFn) error {
 	// merge aliases are defined first, but we only want to traverse them
 	// if we don't find a match on this node first.
 	// traverse them backwards so that the last alias overrides the preceding.
@@ -207,7 +216,7 @@ func (n *navigator) visitAliases(contents []*yaml.Node, head string, tail []stri
 	return nil
 }
 
-func (n *navigator) visitAliasSequence(possibleAliasArray []*yaml.Node, head string, tail []string, pathStack []interface{}, visit mapVisitorFn) error {
+func (n *navigator) visitAliasSequence(possibleAliasArray []*yaml.Node, head string, tail []interface{}, pathStack []interface{}, visit mapVisitorFn) error {
 	// need to search this backwards too, so that aliases defined last override the preceding.
 	for aliasIndex := len(possibleAliasArray) - 1; aliasIndex >= 0; aliasIndex = aliasIndex - 1 {
 		child := possibleAliasArray[aliasIndex]
@@ -223,7 +232,7 @@ func (n *navigator) visitAliasSequence(possibleAliasArray []*yaml.Node, head str
 	return nil
 }
 
-func (n *navigator) splatArray(value *yaml.Node, head string, tail []string, pathStack []interface{}) error {
+func (n *navigator) splatArray(value *yaml.Node, head interface{}, tail []interface{}, pathStack []interface{}) error {
 	for index, childValue := range value.Content {
 		log.Debug("processing")
 		DebugNode(childValue)
@@ -231,6 +240,9 @@ func (n *navigator) splatArray(value *yaml.Node, head string, tail []string, pat
 
 		newPathStack := append(pathStack, index)
 		if n.navigationStrategy.ShouldTraverse(NewNodeContext(childValue, head, tail, newPathStack), childValue.Value) {
+			// here we should not deeply traverse the array if we are appending..not sure how to do that.
+			// need to visit instead...
+			// easiest way is to pop off the head and pass the rest of the tail in.
 			var err = n.doTraverse(childValue, head, tail, newPathStack)
 			if err != nil {
 				return err
@@ -240,14 +252,14 @@ func (n *navigator) splatArray(value *yaml.Node, head string, tail []string, pat
 	return nil
 }
 
-func (n *navigator) appendArray(value *yaml.Node, head string, tail []string, pathStack []interface{}) error {
+func (n *navigator) appendArray(value *yaml.Node, head interface{}, tail []interface{}, pathStack []interface{}) error {
 	var newNode = yaml.Node{Kind: guessKind(head, tail, 0)}
 	value.Content = append(value.Content, &newNode)
 	log.Debug("appending a new node, %v", value.Content)
 	return n.doTraverse(&newNode, head, tail, append(pathStack, len(value.Content)-1))
 }
 
-func (n *navigator) recurseArray(value *yaml.Node, index int64, head string, tail []string, pathStack []interface{}) error {
+func (n *navigator) recurseArray(value *yaml.Node, index int64, head interface{}, tail []interface{}, pathStack []interface{}) error {
 	for int64(len(value.Content)) <= index {
 		value.Content = append(value.Content, &yaml.Node{Kind: guessKind(head, tail, 0)})
 	}

pkg/yqlib/delete_navigation_strategy.go

@@ -1,12 +1,10 @@
 package yqlib
 
 import (
-	"strconv"
-
 	yaml "gopkg.in/yaml.v3"
 )
 
-func DeleteNavigationStrategy(pathElementToDelete string) NavigationStrategy {
+func DeleteNavigationStrategy(pathElementToDelete interface{}) NavigationStrategy {
 	parser := NewPathParser()
 	return &NavigationStrategyImpl{
 		visitedNodes: []*NodeContext{},
@@ -17,6 +15,9 @@ func DeleteNavigationStrategy(pathElementToDelete string) NavigationStrategy {
 		autoCreateMap: func(nodeContext NodeContext) bool {
 			return false
 		},
+		shouldDeeplyTraverse: func(nodeContext NodeContext) bool {
+			return true
+		},
 		visit: func(nodeContext NodeContext) error {
 			node := nodeContext.Node
 			log.Debug("need to find and delete %v in here", pathElementToDelete)
@@ -31,12 +32,12 @@ func DeleteNavigationStrategy(pathElementToDelete string) NavigationStrategy {
 		},
 	}
 }
-func deleteFromMap(pathParser PathParser, contents []*yaml.Node, pathStack []interface{}, pathElementToDelete string) []*yaml.Node {
+func deleteFromMap(pathParser PathParser, contents []*yaml.Node, pathStack []interface{}, pathElementToDelete interface{}) []*yaml.Node {
 	newContents := make([]*yaml.Node, 0)
 	for index := 0; index < len(contents); index = index + 2 {
 		keyNode := contents[index]
 		valueNode := contents[index+1]
-		if !pathParser.MatchesNextPathElement(NewNodeContext(keyNode, pathElementToDelete, []string{}, pathStack), keyNode.Value) {
+		if !pathParser.MatchesNextPathElement(NewNodeContext(keyNode, pathElementToDelete, make([]interface{}, 0), pathStack), keyNode.Value) {
 			log.Debug("adding node %v", keyNode.Value)
 			newContents = append(newContents, keyNode, valueNode)
 		} else {
@@ -46,21 +47,23 @@ func deleteFromMap(pathParser PathParser, contents []*yaml.Node, pathStack []int
 	return newContents
 }
 
-func deleteFromArray(pathParser PathParser, content []*yaml.Node, pathStack []interface{}, pathElementToDelete string) []*yaml.Node {
+func deleteFromArray(pathParser PathParser, content []*yaml.Node, pathStack []interface{}, pathElementToDelete interface{}) []*yaml.Node {
 
-	var indexToDelete, err = strconv.ParseInt(pathElementToDelete, 10, 64) // nolint
+	switch pathElementToDelete := pathElementToDelete.(type) {
-	if err == nil {
+	case int64:
-		return deleteIndexInArray(content, indexToDelete)
+		return deleteIndexInArray(content, pathElementToDelete)
-	}
+	default:
 		log.Debug("%v is not a numeric index, finding matching patterns", pathElementToDelete)
 		var newArray = make([]*yaml.Node, 0)
 
 		for _, childValue := range content {
-		if !pathParser.MatchesNextPathElement(NewNodeContext(childValue, pathElementToDelete, []string{}, pathStack), childValue.Value) {
+			if !pathParser.MatchesNextPathElement(NewNodeContext(childValue, pathElementToDelete, make([]interface{}, 0), pathStack), childValue.Value) {
 				newArray = append(newArray, childValue)
 			}
 		}
 		return newArray
+	}
+
 }
 
 func deleteIndexInArray(content []*yaml.Node, index int64) []*yaml.Node {

pkg/yqlib/encoder.go

@@ -1,6 +1,7 @@
 package yqlib
 
 import (
+	"bytes"
 	"encoding/json"
 	"io"
 
@@ -12,20 +13,45 @@ type Encoder interface {
 }
 
 type yamlEncoder struct {
-	encoder *yaml.Encoder
+	destination io.Writer
+	indent      int
+	colorise    bool
+	firstDoc    bool
 }
 
-func NewYamlEncoder(destination io.Writer, indent int) Encoder {
+func NewYamlEncoder(destination io.Writer, indent int, colorise bool) Encoder {
-	var encoder = yaml.NewEncoder(destination)
 	if indent < 0 {
 		indent = 0
 	}
-	encoder.SetIndent(indent)
+	return &yamlEncoder{destination, indent, colorise, true}
-	return &yamlEncoder{encoder}
 }
 
 func (ye *yamlEncoder) Encode(node *yaml.Node) error {
-	return ye.encoder.Encode(node)
+
+	destination := ye.destination
+	tempBuffer := bytes.NewBuffer(nil)
+	if ye.colorise {
+		destination = tempBuffer
+	}
+
+	var encoder = yaml.NewEncoder(destination)
+
+	encoder.SetIndent(ye.indent)
+	// TODO: work out if the first doc had a separator or not.
+	if ye.firstDoc {
+		ye.firstDoc = false
+	} else if _, err := destination.Write([]byte("---\n")); err != nil {
+		return err
+	}
+
+	if err := encoder.Encode(node); err != nil {
+		return err
+	}
+
+	if ye.colorise {
+		return ColorizeAndPrint(tempBuffer.Bytes(), ye.destination)
+	}
+	return nil
 }
 
 type jsonEncoder struct {

pkg/yqlib/lib.go

@@ -19,6 +19,23 @@ type UpdateCommand struct {
 	Overwrite bool
 }
 
+func KindString(kind yaml.Kind) string {
+	switch kind {
+	case yaml.ScalarNode:
+		return "ScalarNode"
+	case yaml.SequenceNode:
+		return "SequenceNode"
+	case yaml.MappingNode:
+		return "MappingNode"
+	case yaml.DocumentNode:
+		return "DocumentNode"
+	case yaml.AliasNode:
+		return "AliasNode"
+	default:
+		return "unknown!"
+	}
+}
+
 func DebugNode(value *yaml.Node) {
 	if value == nil {
 		log.Debug("-- node is nil --")
@@ -30,7 +47,7 @@ func DebugNode(value *yaml.Node) {
 			log.Error("Error debugging node, %v", errorEncoding.Error())
 		}
 		encoder.Close()
-		log.Debug("Tag: %v", value.Tag)
+		log.Debug("Tag: %v, Kind: %v", value.Tag, KindString(value.Kind))
 		log.Debug("%v", buf.String())
 	}
 }
@@ -43,7 +60,7 @@ func mergePathStackToString(pathStack []interface{}, appendArrays bool) string {
 	var sb strings.Builder
 	for index, path := range pathStack {
 		switch path.(type) {
-		case int:
+		case int, int64:
 			if appendArrays {
 				sb.WriteString("[+]")
 			} else {
@@ -52,13 +69,22 @@ func mergePathStackToString(pathStack []interface{}, appendArrays bool) string {
 
 		default:
 			s := fmt.Sprintf("%v", path)
-			hasDot := strings.Contains(s, ".")
+			var _, errParsingInt = strconv.ParseInt(s, 10, 64) // nolint
-			if hasDot {
+
-				sb.WriteString("[")
+			hasSpecial := strings.Contains(s, ".") || strings.Contains(s, "[") || strings.Contains(s, "]") || strings.Contains(s, "\"")
+			hasDoubleQuotes := strings.Contains(s, "\"")
+			wrappingCharacterStart := "\""
+			wrappingCharacterEnd := "\""
+			if hasDoubleQuotes {
+				wrappingCharacterStart = "("
+				wrappingCharacterEnd = ")"
+			}
+			if hasSpecial || errParsingInt == nil {
+				sb.WriteString(wrappingCharacterStart)
 			}
 			sb.WriteString(s)
-			if hasDot {
+			if hasSpecial || errParsingInt == nil {
-				sb.WriteString("]")
+				sb.WriteString(wrappingCharacterEnd)
 			}
 		}
 
@@ -66,38 +92,47 @@ func mergePathStackToString(pathStack []interface{}, appendArrays bool) string {
 			sb.WriteString(".")
 		}
 	}
-	return sb.String()
+	var pathString = sb.String()
+	log.Debug("got a path string: %v", pathString)
+	return pathString
 }
 
-func guessKind(head string, tail []string, guess yaml.Kind) yaml.Kind {
+func guessKind(head interface{}, tail []interface{}, guess yaml.Kind) yaml.Kind {
-	log.Debug("tail %v", tail)
+	log.Debug("guessKind: tail %v", tail)
 	if len(tail) == 0 && guess == 0 {
 		log.Debug("end of path, must be a scalar")
 		return yaml.ScalarNode
 	} else if len(tail) == 0 {
 		return guess
 	}
-
+	var next = tail[0]
-	var _, errorParsingInt = strconv.ParseInt(tail[0], 10, 64)
+	switch next.(type) {
-	if tail[0] == "+" || errorParsingInt == nil {
+	case int64:
+		return yaml.SequenceNode
+	default:
+		var nextString = fmt.Sprintf("%v", next)
+		if nextString == "+" {
 			return yaml.SequenceNode
 		}
 		pathParser := NewPathParser()
-	if (pathParser.IsPathExpression(tail[0]) || head == "**") && (guess == yaml.SequenceNode || guess == yaml.MappingNode) {
+		if pathParser.IsPathExpression(nextString) && (guess == yaml.SequenceNode || guess == yaml.MappingNode) {
 			return guess
-	}
+		} else if guess == yaml.AliasNode {
-	if guess == yaml.AliasNode {
 			log.Debug("guess was an alias, okey doke.")
 			return guess
+		} else if head == "**" {
+			log.Debug("deep wildcard, go with the guess")
+			return guess
 		}
 		log.Debug("forcing a mapping node")
 		log.Debug("yaml.SequenceNode %v", guess == yaml.SequenceNode)
 		log.Debug("yaml.ScalarNode %v", guess == yaml.ScalarNode)
 		return yaml.MappingNode
+	}
 }
 
 type YqLib interface {
-	Get(rootNode *yaml.Node, path string) ([]*NodeContext, error)
+	Get(rootNode *yaml.Node, path string, deeplyTraverseArrays bool) ([]*NodeContext, error)
 	Update(rootNode *yaml.Node, updateCommand UpdateCommand, autoCreate bool) error
 	New(path string) yaml.Node
 
@@ -115,9 +150,9 @@ func NewYqLib() YqLib {
 	}
 }
 
-func (l *lib) Get(rootNode *yaml.Node, path string) ([]*NodeContext, error) {
+func (l *lib) Get(rootNode *yaml.Node, path string, deeplyTraverseArrays bool) ([]*NodeContext, error) {
 	var paths = l.parser.ParsePath(path)
-	navigationStrategy := ReadNavigationStrategy()
+	navigationStrategy := ReadNavigationStrategy(deeplyTraverseArrays)
 	navigator := NewDataNavigator(navigationStrategy)
 	error := navigator.Traverse(rootNode, paths)
 	return navigationStrategy.GetVisitedNodes(), error

pkg/yqlib/navigation_strategy.go

@@ -8,13 +8,13 @@ import (
 
 type NodeContext struct {
 	Node      *yaml.Node
-	Head      string
+	Head      interface{}
-	Tail      []string
+	Tail      []interface{}
 	PathStack []interface{}
 }
 
-func NewNodeContext(node *yaml.Node, head string, tail []string, pathStack []interface{}) NodeContext {
+func NewNodeContext(node *yaml.Node, head interface{}, tail []interface{}, pathStack []interface{}) NodeContext {
-	newTail := make([]string, len(tail))
+	newTail := make([]interface{}, len(tail))
 	copy(newTail, tail)
 
 	newPathStack := make([]interface{}, len(pathStack))
@@ -34,6 +34,7 @@ type NavigationStrategy interface {
 	// node key is the string value of the last element in the path stack
 	// we use it to match against the pathExpression in head.
 	ShouldTraverse(nodeContext NodeContext, nodeKey string) bool
+	ShouldDeeplyTraverse(nodeContext NodeContext) bool
 	GetVisitedNodes() []*NodeContext
 	DebugVisitedNodes()
 	GetPathParser() PathParser
@@ -44,6 +45,7 @@ type NavigationStrategyImpl struct {
 	autoCreateMap        func(nodeContext NodeContext) bool
 	visit                func(nodeContext NodeContext) error
 	shouldVisitExtraFn   func(nodeContext NodeContext) bool
+	shouldDeeplyTraverse func(nodeContext NodeContext) bool
 	visitedNodes         []*NodeContext
 	pathParser           PathParser
 }
@@ -64,6 +66,10 @@ func (ns *NavigationStrategyImpl) AutoCreateMap(nodeContext NodeContext) bool {
 	return ns.autoCreateMap(nodeContext)
 }
 
+func (ns *NavigationStrategyImpl) ShouldDeeplyTraverse(nodeContext NodeContext) bool {
+	return ns.shouldDeeplyTraverse(nodeContext)
+}
+
 func (ns *NavigationStrategyImpl) ShouldTraverse(nodeContext NodeContext, nodeKey string) bool {
 	// we should traverse aliases (if enabled), but not visit them :/
 	if len(nodeContext.PathStack) == 0 {
@@ -84,7 +90,6 @@ func (ns *NavigationStrategyImpl) shouldVisit(nodeContext NodeContext) bool {
 		return true
 	}
 	log.Debug("tail len %v", len(nodeContext.Tail))
-	// SOMETHING HERE!
 
 	if ns.alreadyVisited(pathStack) || len(nodeContext.Tail) != 0 {
 		return false

pkg/yqlib/path_parser.go

@@ -1,12 +1,13 @@
 package yqlib
 
 import (
+	"fmt"
 	"strconv"
 	"strings"
 )
 
 type PathParser interface {
-	ParsePath(path string) []string
+	ParsePath(path string) []interface{}
 	MatchesNextPathElement(nodeContext NodeContext, nodeKey string) bool
 	IsPathExpression(pathElement string) bool
 }
@@ -42,9 +43,11 @@ func (p *pathParser) MatchesNextPathElement(nodeContext NodeContext, nodeKey str
 	if head == "**" || head == "*" {
 		return true
 	}
-	if strings.Contains(head, "==") {
+	var headString = fmt.Sprintf("%v", head)
+
+	if strings.Contains(headString, "==") {
 		log.Debug("ooh deep recursion time")
-		result := strings.SplitN(head, "==", 2)
+		result := strings.SplitN(headString, "==", 2)
 		path := strings.TrimSpace(result[0])
 		value := strings.TrimSpace(result[1])
 		log.Debug("path %v", path)
@@ -70,17 +73,18 @@ func (p *pathParser) MatchesNextPathElement(nodeContext NodeContext, nodeKey str
 		}
 	}
 
-	return matchesString(head, nodeKey)
+	return matchesString(headString, nodeKey)
 }
 
-func (p *pathParser) ParsePath(path string) []string {
+func (p *pathParser) ParsePath(path string) []interface{} {
+	var paths = make([]interface{}, 0)
 	if path == "" {
-		return []string{}
+		return paths
 	}
-	return p.parsePathAccum([]string{}, path)
+	return p.parsePathAccum(paths, path)
 }
 
-func (p *pathParser) parsePathAccum(paths []string, remaining string) []string {
+func (p *pathParser) parsePathAccum(paths []interface{}, remaining string) []interface{} {
 	head, tail := p.nextYamlPath(remaining)
 	if tail == "" {
 		return append(paths, head)
@@ -88,11 +92,16 @@ func (p *pathParser) parsePathAccum(paths []string, remaining string) []string {
 	return p.parsePathAccum(append(paths, head), tail)
 }
 
-func (p *pathParser) nextYamlPath(path string) (pathElement string, remaining string) {
+func (p *pathParser) nextYamlPath(path string) (pathElement interface{}, remaining string) {
 	switch path[0] {
 	case '[':
 		// e.g [0].blah.cat -> we need to return "0" and "blah.cat"
-		return p.search(path[1:], []uint8{']'}, true)
+		var value, remainingBit = p.search(path[1:], []uint8{']'}, true)
+		var number, errParsingInt = strconv.ParseInt(value, 10, 64) // nolint
+		if errParsingInt == nil {
+			return number, remainingBit
+		}
+		return value, remainingBit
 	case '"':
 		// e.g "a.b".blah.cat -> we need to return "a.b" and "blah.cat"
 		return p.search(path[1:], []uint8{'"'}, true)

pkg/yqlib/path_parser_test.go

@@ -10,20 +10,21 @@ var parser = NewPathParser()
 
 var parsePathsTests = []struct {
 	path          string
-	expectedPaths []string
+	expectedPaths []interface{}
 }{
-	{"a.b", []string{"a", "b"}},
+	{"a.b", append(make([]interface{}, 0), "a", "b")},
-	{"a.b.**", []string{"a", "b", "**"}},
+	{"a.b.**", append(make([]interface{}, 0), "a", "b", "**")},
-	{"a.b.*", []string{"a", "b", "*"}},
+	{"a.b.*", append(make([]interface{}, 0), "a", "b", "*")},
-	{"a.b[0]", []string{"a", "b", "0"}},
+	{"a.b[0]", append(make([]interface{}, 0), "a", "b", int64(0))},
-	{"a.b.d[+]", []string{"a", "b", "d", "+"}},
+	{"a.b.0", append(make([]interface{}, 0), "a", "b", "0")},
-	{"a", []string{"a"}},
+	{"a.b.d[+]", append(make([]interface{}, 0), "a", "b", "d", "+")},
-	{"a.b.c", []string{"a", "b", "c"}},
+	{"a", append(make([]interface{}, 0), "a")},
-	{"\"a.b\".c", []string{"a.b", "c"}},
+	{"a.b.c", append(make([]interface{}, 0), "a", "b", "c")},
-	{"a.\"b.c\".d", []string{"a", "b.c", "d"}},
+	{"\"a.b\".c", append(make([]interface{}, 0), "a.b", "c")},
-	{"[1].a.d", []string{"1", "a", "d"}},
+	{"a.\"b.c\".d", append(make([]interface{}, 0), "a", "b.c", "d")},
-	{"a[0].c", []string{"a", "0", "c"}},
+	{"[1].a.d", append(make([]interface{}, 0), int64(1), "a", "d")},
-	{"[0]", []string{"0"}},
+	{"a[0].c", append(make([]interface{}, 0), "a", int64(0), "c")},
+	{"[0]", append(make([]interface{}, 0), int64(0))},
 }
 
 func TestPathParserParsePath(t *testing.T) {

pkg/yqlib/read_navigation_strategy.go

@@ -1,6 +1,6 @@
 package yqlib
 
-func ReadNavigationStrategy() NavigationStrategy {
+func ReadNavigationStrategy(deeplyTraverseArrays bool) NavigationStrategy {
 	return &NavigationStrategyImpl{
 		visitedNodes: []*NodeContext{},
 		pathParser:   NewPathParser(),
@@ -13,5 +13,18 @@ func ReadNavigationStrategy() NavigationStrategy {
 		visit: func(nodeContext NodeContext) error {
 			return nil
 		},
+		shouldDeeplyTraverse: func(nodeContext NodeContext) bool {
+			var isInArray = false
+			if len(nodeContext.PathStack) > 0 {
+				var lastElement = nodeContext.PathStack[len(nodeContext.PathStack)-1]
+				switch lastElement.(type) {
+				case int:
+					isInArray = true
+				default:
+					isInArray = false
+				}
+			}
+			return deeplyTraverseArrays || !isInArray
+		},
 	}
 }

pkg/yqlib/update_navigation_strategy.go

@@ -10,6 +10,9 @@ func UpdateNavigationStrategy(updateCommand UpdateCommand, autoCreate bool) Navi
 		autoCreateMap: func(nodeContext NodeContext) bool {
 			return autoCreate
 		},
+		shouldDeeplyTraverse: func(nodeContext NodeContext) bool {
+			return true
+		},
 		visit: func(nodeContext NodeContext) error {
 			node := nodeContext.Node
 			changesToApply := updateCommand.Value

release_instructions.txt

@@ -33,6 +33,8 @@
   - docker build .  -t mikefarah/yq:latest -t mikefarah/yq:VERSION 
 
 - debian package
+  - ensure you get all vendor dependencies before packaging
+	```go mod vendor```
   - execute 
     ```dch -i```
   - fill debian/changelog with changes from last version

scripts/format.sh

@@ -1,3 +1,4 @@
 #!/bin/bash
 
 find . \( -path ./vendor \) -prune -o -name "*.go" -exec goimports -w {} \;
+go mod tidy

scripts/publish.sh

@@ -32,5 +32,5 @@ upload() {
     done < <(find ./build -mindepth 1 -maxdepth 1 -print0)
 }
 
-# release
+release
 upload

snap/snapcraft.yaml

@@ -1,5 +1,5 @@
 name: yq
-version: '3.1.0'
+version: '3.1.2'
 summary: A lightweight and portable command-line YAML processor
 description: |
   The aim of the project is to be the jq or sed of yaml files.

test.yml

@@ -1,3 +0,0 @@
-b:
-  c: thing
-  d: another thing