Bumping version to beta2

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,8 +1,8 @@
 ---
-name: Bug report - V3
+name: Bug report
 about: Create a report to help us improve
 title: ''
-labels: bug, v3
+labels: bug
 assignees: ''
 
 ---
@@ -10,12 +10,11 @@ assignees: ''
 **Describe the bug**
 A clear and concise description of what the bug is.
 
-Version of yq: 3.X.X
+version of yq: 
-Operating system: mac/linux/windows/....
+operating system:
-Installed via: docker/binary release/homebrew/snap/...
 
 **Input Yaml**
-Concise yaml document(s) (as simple as possible to show the bug, please keep it to 10 lines or less)
+Concise yaml document(s) (as simple as possible to show the bug)
 data1.yml:
 ```yaml
 this: should really work

.github/ISSUE_TEMPLATE/bug_report_v4.md

@@ -1,49 +0,0 @@
----
-name: Bug report - V4
-about: Create a report to help us improve
-title: ''
-labels: bug, v4
-assignees: ''
-
----
-
-**Describe the bug**
-A clear and concise description of what the bug is.
-
-Version of yq: 4.X.X
-Operating system: mac/linux/windows/....
-Installed via: docker/binary release/homebrew/snap/...
-
-**Input Yaml**
-Concise yaml document(s) (as simple as possible to show the bug, please keep it to 10 lines or less)
-data1.yml:
-```yaml
-this: should really work
-```
-
-data2.yml:
-```yaml
-but: it strangely didn't
-```
-
-**Command**
-The command you ran:
-```
-yq eval-all 'select(fileIndex==0) | .a.b.c' data1.yml data2.yml
-```
-
-**Actual behavior**
-
-```yaml
-cat: meow
-```
-
-**Expected behavior**
-
-```yaml
-this: should really work
-but: it strangely didn't
-```
-
-**Additional context**
-Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,21 +1,17 @@
 ---
-name: Feature request - V4
+name: Feature request
 about: Suggest an idea for this project
 title: ''
-labels: enhancement, v4
+labels: enhancement
 assignees: ''
 
 ---
 
-**Please describe your feature request.**
+**Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the request is and what it would solve. 
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
-Ex. I wish I could use yq to [...]
-
-Please note that V3 will no longer have any enhancements.
 
 **Describe the solution you'd like**
 If we have data1.yml like:
-(please keep to around 10 lines )
 
 ```yaml
 country: Australia
@@ -24,7 +20,7 @@ country: Australia
 And we run a command:
 
 ```bash
-yq eval 'predictWeatherOf(.country)'
+yq predictWeather data1.yml
 ```
 
 it could output

.github/workflows/go.yml

@@ -28,4 +28,7 @@ jobs:
       run: |
         export PATH=${PATH}:`go env GOPATH`/bin
         scripts/devtools.sh
+    - name: Build
+      run: |
+        export PATH=${PATH}:`go env GOPATH`/bin
         make local build

.github/workflows/release.yml

@@ -1,48 +0,0 @@
-name: Publish image to Dockerhub
-
-on:
-  release:
-    types: [released]
-
-jobs:
-  build:
-    env:
-      IMAGE_NAME: mikefarah/yq
-    runs-on: ubuntu-latest
-    steps:
-      - name: Get latest release tag      
-        uses: oprypin/find-latest-tag@v1
-        with:
-          repository: mikefarah/yq  # The repository to scan.
-          releases-only: true  # We know that all relevant tags have a GitHub release for them.
-        id: yq  
-        
-      - name: Clone source code
-        uses: actions/checkout@v2
-        with:
-          ref: ${{ steps.yq.outputs.tag }}
-      
-      - name: Set up QEMU
-        uses: docker/setup-qemu-action@v1
-        with:
-          platforms: all
-  
-      - name: Set up Docker Buildx
-        id: buildx
-        uses: docker/setup-buildx-action@v1
-        with:
-          version: latest
-
-      - name: Available platforms
-        run: echo ${{ steps.buildx.outputs.platforms }} && docker version
- 
-      - name: Build and push image
-        run: |
-          IMAGE_VERSION="$(git describe --tags --abbrev=0)"
-          SHORT_SHA1=$(git rev-parse --short HEAD)
-          PLATFORMS="linux/amd64,linux/ppc64le,linux/arm64"
-          echo "Building and pushing version ${IMAGE_VERSION} of image ${IMAGE_NAME}"
-          echo ${{ secrets.DOCKER_PASSWORD }} | docker login -u ${{ secrets.DOCKER_USERNAME }} --password-stdin
-          docker buildx build --platform "${PLATFORMS}" -t "${IMAGE_NAME}:${IMAGE_VERSION}"  -t "${IMAGE_NAME}:latest" \
-            --push .
-    

README.md

@@ -12,7 +12,6 @@ The aim of the project is to be the [jq](https://github.com/stedolan/jq) or sed
 ### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)
 
 ### MacOS:
-Using [Homebrew](https://brew.sh/)
 ```
 brew install yq
 ```
@@ -45,12 +44,13 @@ rm /etc/myfile.tmp
 Use wget to download the pre-compiled binaries:
 
 ```bash
-wget https://github.com/mikefarah/yq/releases/download/${VERSION}/${BINARY} -O /usr/bin/yq &&\
+wget https://github.com/mikefarah/yq/releases/download/{VERSION}/{BINARY} -O /usr/bin/yq &&\
     chmod +x /usr/bin/yq
 ```
 
 For instance, VERSION=4.0.0 and BINARY=yq_linux_amd64
 
+
 ### Run with Docker
 
 #### Oneshot use:
@@ -88,14 +88,6 @@ choco install yq
 ```
 Supported by @chillum (https://chocolatey.org/packages/yq)
 
-### Mac:
-Using [MacPorts](https://www.macports.org/)
-```
-sudo port selfupdate
-sudo port install yq
-```
-Supported by @herbygillot (https://ports.macports.org/maintainer/github/herbygillot)
-
 ### Alpine Linux
 - Enable edge/community repo by adding ```$MIRROR/alpine/edge/community``` to ```/etc/apk/repositories```
 - Update database index with ```apk update```
@@ -116,18 +108,21 @@ Supported by @rmescandon (https://launchpad.net/~rmescandon/+archive/ubuntu/yq)
 
 ## Features
 - Written in portable go, so you can download a lovely dependency free binary
-- Uses similar syntax as `jq` but works with YAML and JSON files
-- Fully supports multi document yaml files
 - Colorized yaml output
-- [Deeply traverse yaml](https://mikefarah.gitbook.io/yq/v/v4.x/traverse)
+- [Deep read a yaml file with a given path expression](https://mikefarah.gitbook.io/yq/v/v4.x/traverse)
-- [Sort yaml by keys](https://mikefarah.gitbook.io/yq/v/v4.x/sort-keys)
+- [Return the lengths of arrays/object/scalars](https://mikefarah.gitbook.io/yq/commands/read#printing-length-of-the-results)
-- Manipulate yaml [comments](https://app.gitbook.com/@mikefarah/s/yq/v/v4.x/comment-operators), [styling](https://app.gitbook.com/@mikefarah/s/yq/v/v4.x/style), [tags](https://app.gitbook.com/@mikefarah/s/yq/v/v4.x/tag) and [anchors](https://app.gitbook.com/@mikefarah/s/yq/v/v4.x/explode).
+- Update a yaml file given a [path expression](https://mikefarah.gitbook.io/yq/commands/write-update#basic) or [script file](https://mikefarah.gitbook.io/yq/commands/write-update#basic)
-- [Update yaml inplace](https://mikefarah.gitbook.io/yq/v/v4.x/commands/evaluate#flags)
+- Update creates any missing entries in the path on the fly
-- [Complex expressions to select and update](https://mikefarah.gitbook.io/yq/v/v4.x/select#select-and-update-matching-values-in-map)
+- Deeply [compare](https://mikefarah.gitbook.io/yq/commands/compare) yaml files
-- Keeps yaml formatting and comments when updating (though there are issues with whitespace)
+- Keeps yaml formatting and comments when updating
-- [Convert to/from json to yaml](https://mikefarah.gitbook.io/yq/v/v4.x/usage/convert)
+- [Validate a yaml file](https://mikefarah.gitbook.io/yq/commands/validate)
-- [Pipe data in by using '-'](https://mikefarah.gitbook.io/yq/v/v4.x/commands/evaluate)
+- Create a yaml file given a [deep path and value](https://mikefarah.gitbook.io/yq/commands/create#creating-a-simple-yaml-file) or a [script file](https://mikefarah.gitbook.io/yq/commands/create#creating-using-a-create-script)
-- General shell completion scripts (bash/zsh/fish/powershell) (https://mikefarah.gitbook.io/yq/v/v4.x/commands/shell-completion)
+- [Prefix a path to a yaml file](https://mikefarah.gitbook.io/yq/commands/prefix)
+- [Convert to/from json to yaml](https://mikefarah.gitbook.io/yq/usage/convert)
+- [Pipe data in by using '-'](https://mikefarah.gitbook.io/yq/commands/read#from-stdin)
+- [Merge](https://mikefarah.gitbook.io/yq/commands/merge) multiple yaml files with various options for [overriding](https://mikefarah.gitbook.io/yq/commands/merge#overwrite-values) and [appending](https://mikefarah.gitbook.io/yq/commands/merge#append-values-with-arrays)
+- Supports multiple documents in a single yaml file for [reading](https://mikefarah.gitbook.io/yq/commands/read#multiple-documents), [writing](https://mikefarah.gitbook.io/yq/commands/write-update#multiple-documents) and [merging](https://mikefarah.gitbook.io/yq/commands/merge#multiple-documents)
+- General shell completion scripts (bash/zsh/fish/powershell) (https://mikefarah.gitbook.io/yq/commands/shell-completion)
 
 ## [Usage](https://mikefarah.gitbook.io/yq/)
 
@@ -139,42 +134,32 @@ Usage:
   yq [command]
 
 Available Commands:
-  eval             Apply expression to each document in each yaml file given in sequence
+  compare          yq x [--prettyPrint/-P] dataA.yaml dataB.yaml 'b.e(name==fr*).value'
-  eval-all         Loads _all_ yaml documents of _all_ yaml files and runs expression once
+  delete           yq d [--inplace/-i] [--doc/-d index] sample.yaml 'b.e(name==fred)'
   help             Help about any command
-  shell-completion Generate completion script
+  merge            yq m [--inplace/-i] [--doc/-d index] [--overwrite/-x] [--append/-a] sample.yaml sample2.yaml
+  new              yq n [--script/-s script_file] a.b.c newValue
+  prefix           yq p [--inplace/-i] [--doc/-d index] sample.yaml a.b.c
+  read             yq r [--printMode/-p pv] sample.yaml 'b.e(name==fr*).value'
+  shell-completion Generates shell completion scripts
+  validate         yq v sample.yaml
+  write            yq w [--inplace/-i] [--script/-s script_file] [--doc/-d index] sample.yaml 'b.e(name==fr*).value' newValue
 
 Flags:
-  -C, --colors        force print with colors
+  -C, --colors        print with colors
-  -e, --exit-status   set exit status if there are no matches or null or false is returned
   -h, --help          help for yq
   -I, --indent int    sets indent level for output (default 2)
-  -i, --inplace       update the yaml file inplace of first yaml file given.
+  -P, --prettyPrint   pretty print
-  -M, --no-colors     force print with no colors
+  -j, --tojson        output as json. By default it prints a json document in one line, use the prettyPrint flag to print a formatted doc.
-  -N, --no-doc        Don't print document separators (---)
-  -n, --null-input    Don't read input, simply evaluate the expression given. Useful for creating yaml docs from scratch.
-  -j, --tojson        output as json. Set indent to 0 to print json in one line.
   -v, --verbose       verbose mode
   -V, --version       Print version information and quit
 
 Use "yq [command] --help" for more information about a command.
 ```
 
-Simple Example:
+## Upgrade from V2
-
+If you've been using v2 and want/need to upgrade, checkout the [upgrade guide](https://mikefarah.gitbook.io/yq/upgrading-from-v2).
-```bash
-yq e '.a.b | length' f1.yml f2.yml 
-```
-
-## Upgrade from V3
-If you've been using v3 and want/need to upgrade, checkout the [upgrade guide](https://mikefarah.gitbook.io/yq/v/v4.x/upgrading-from-v3).
-
-## V4 is in development!
-If you're keen - check out the alpha release [here](https://github.com/mikefarah/yq/releases/), or in docker `mikefarah/yq:4-beta1` and the docs (also in beta) [here](https://mikefarah.gitbook.io/yq/v/v4.x-alpha/).
-
-V4 is quite different from V3 (sorry for the migration), however it will be much more similar to ```jq```, use a similar expression syntax and therefore support much more complex functionality! 
-
-For now - new features will be held off from V3 in anticipation of the V4 build. Critical fixes / issues will still be released.
 
 ## Known Issues / Missing Features
 - `yq` attempts to preserve comment positions and whitespace as much as possible, but it does not handle all scenarios (see https://github.com/go-yaml/yaml/tree/v3 for details)
+- You cannot (yet) select multiple paths/keys from the yaml to be printed out (https://github.com/mikefarah/yq/issues/287)

cmd/version.go

@@ -11,7 +11,7 @@ var (
 	GitDescribe string
 
 	// Version is main version number that is being run at the moment.
-	Version = "4.0.0"
+	Version = "4.0.0-beta2"
 
 	// VersionPrerelease is a pre-release marker for the version. If this is "" (empty string)
 	// then it means that it is a final release. Otherwise, this is a pre-release

go.sum

@@ -263,7 +263,6 @@ golang.org/x/sys v0.0.0-20191010194322-b09406accb47/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20191026070338-33540a1f6037 h1:YyJpGZS1sBuBCzLAR1VEpK193GlqGZbnPFnPV/5Rsb4=
 golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae h1:/WDfKMnPU+m5M4xB+6x4kaepxRw6jWvR5iDRdvjHgy8=
 golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f h1:+Nyd8tzPX9R7BWHguqsrbFdRx3WQ/1ib8I44HXV5yTA=
 golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=

pkg/yqlib/path_tokeniser.go

@@ -1,7 +1,6 @@
 package yqlib
 
 import (
-	"fmt"
 	"strconv"
 
 	lex "github.com/timtadh/lexmachine"
@@ -289,7 +288,7 @@ func (p *pathTokeniser) Tokenise(path string) ([]*Token, error) {
 	scanner, err := p.lexer.Scanner([]byte(path))
 
 	if err != nil {
-		return nil, fmt.Errorf("Parsing expression: %v", err)
+		return nil, err
 	}
 	var tokens []*Token
 	for tok, err, eof := scanner.Next(); !eof; tok, err, eof = scanner.Next() {
@@ -300,7 +299,7 @@ func (p *pathTokeniser) Tokenise(path string) ([]*Token, error) {
 			tokens = append(tokens, token)
 		}
 		if err != nil {
-			return nil, fmt.Errorf("Parsing expression: %v", err)
+			return nil, err
 		}
 	}
 	var postProcessedTokens = make([]*Token, 0)

pkg/yqlib/path_tree.go

@@ -1,9 +1,6 @@
 package yqlib
 
-import (
+import "fmt"
-	"fmt"
-	"strings"
-)
 
 var myPathTokeniser = NewPathTokeniser()
 var myPathPostfixer = NewPathPostFixer()
@@ -52,16 +49,10 @@ func (p *pathTreeCreator) CreatePathTree(postFixPath []*Operation) (*PathTreeNod
 		if Operation.OperationType.NumArgs > 0 {
 			numArgs := Operation.OperationType.NumArgs
 			if numArgs == 1 {
-				if len(stack) < 1 {
-					return nil, fmt.Errorf("'%v' expects 1 arg but received none", strings.TrimSpace(Operation.StringValue))
-				}
 				remaining, rhs := stack[:len(stack)-1], stack[len(stack)-1]
 				newNode.Rhs = rhs
 				stack = remaining
 			} else if numArgs == 2 {
-				if len(stack) < 2 {
-					return nil, fmt.Errorf("'%v' expects 2 args but there is %v", strings.TrimSpace(Operation.StringValue), len(stack))
-				}
 				remaining, lhs, rhs := stack[:len(stack)-2], stack[len(stack)-2], stack[len(stack)-1]
 				newNode.Lhs = lhs
 				newNode.Rhs = rhs
@@ -71,7 +62,7 @@ func (p *pathTreeCreator) CreatePathTree(postFixPath []*Operation) (*PathTreeNod
 		stack = append(stack, &newNode)
 	}
 	if len(stack) != 1 {
-		return nil, fmt.Errorf("expected end of expression but found '%v', please check expression syntax", strings.TrimSpace(stack[1].Operation.StringValue))
+		return nil, fmt.Errorf("expected stack to have 1 thing but its %v", stack)
 	}
 	return stack[0], nil
 }

pkg/yqlib/path_tree_test.go

@@ -1,42 +0,0 @@
-package yqlib
-
-import (
-	"testing"
-
-	"github.com/mikefarah/yq/v4/test"
-)
-
-func TestPathTreeNoArgsForTwoArgOp(t *testing.T) {
-	_, err := treeCreator.ParsePath("=")
-	test.AssertResultComplex(t, "'=' expects 2 args but there is 0", err.Error())
-}
-
-func TestPathTreeOneLhsArgsForTwoArgOp(t *testing.T) {
-	_, err := treeCreator.ParsePath(".a =")
-	test.AssertResultComplex(t, "'=' expects 2 args but there is 1", err.Error())
-}
-
-func TestPathTreeOneRhsArgsForTwoArgOp(t *testing.T) {
-	_, err := treeCreator.ParsePath("= .a")
-	test.AssertResultComplex(t, "'=' expects 2 args but there is 1", err.Error())
-}
-
-func TestPathTreeTwoArgsForTwoArgOp(t *testing.T) {
-	_, err := treeCreator.ParsePath(".a = .b")
-	test.AssertResultComplex(t, nil, err)
-}
-
-func TestPathTreeNoArgsForOneArgOp(t *testing.T) {
-	_, err := treeCreator.ParsePath("explode")
-	test.AssertResultComplex(t, "'explode' expects 1 arg but received none", err.Error())
-}
-
-func TestPathTreeOneArgForOneArgOp(t *testing.T) {
-	_, err := treeCreator.ParsePath("explode(.)")
-	test.AssertResultComplex(t, nil, err)
-}
-
-func TestPathTreeExtraArgs(t *testing.T) {
-	_, err := treeCreator.ParsePath("sortKeys(.) explode(.)")
-	test.AssertResultComplex(t, "expected end of expression but found 'explode', please check expression syntax", err.Error())
-}

pkg/yqlib/printer.go

@@ -22,7 +22,6 @@ type resultsPrinter struct {
 	writer             io.Writer
 	firstTimePrinting  bool
 	previousDocIndex   uint
-	previousFileIndex  int
 	printedMatches     bool
 }
 
@@ -90,20 +89,19 @@ func (p *resultsPrinter) PrintResults(matchingNodes *list.List) error {
 		return nil
 	}
 	if p.firstTimePrinting {
-		node := matchingNodes.Front().Value.(*CandidateNode)
+		p.previousDocIndex = matchingNodes.Front().Value.(*CandidateNode).Document
-		p.previousDocIndex = node.Document
-		p.previousFileIndex = node.FileIndex
 		p.firstTimePrinting = false
 	}
 
 	for el := matchingNodes.Front(); el != nil; el = el.Next() {
 		mappedDoc := el.Value.(*CandidateNode)
 		log.Debug("-- print sep logic: p.firstTimePrinting: %v, previousDocIndex: %v, mappedDoc.Document: %v, printDocSeparators: %v", p.firstTimePrinting, p.previousDocIndex, mappedDoc.Document, p.printDocSeparators)
-		if (p.previousDocIndex != mappedDoc.Document || p.previousFileIndex != mappedDoc.FileIndex) && p.printDocSeparators {
+		if (p.previousDocIndex != mappedDoc.Document) && p.printDocSeparators {
 			log.Debug("-- writing doc sep")
 			if err := p.writeString(bufferedWriter, "---\n"); err != nil {
 				return err
 			}
+
 		}
 
 		if err := p.printNode(mappedDoc.Node, bufferedWriter); err != nil {

pkg/yqlib/printer_test.go

@@ -52,53 +52,7 @@ func TestPrinterMultipleDocsInSequence(t *testing.T) {
 
 	writer.Flush()
 	test.AssertResult(t, multiDocSample, output.String())
-}
 
-func TestPrinterMultipleFilesInSequence(t *testing.T) {
-	var output bytes.Buffer
-	var writer = bufio.NewWriter(&output)
-	printer := NewPrinter(writer, false, true, false, 2, true)
-
-	inputs, err := readDocuments(strings.NewReader(multiDocSample), "sample.yml", 0)
-	if err != nil {
-		panic(err)
-	}
-
-	el := inputs.Front()
-	elNode := el.Value.(*CandidateNode)
-	elNode.Document = 0
-	elNode.FileIndex = 0
-	sample1 := nodeToMap(elNode)
-
-	el = el.Next()
-	elNode = el.Value.(*CandidateNode)
-	elNode.Document = 0
-	elNode.FileIndex = 1
-	sample2 := nodeToMap(elNode)
-
-	el = el.Next()
-	elNode = el.Value.(*CandidateNode)
-	elNode.Document = 0
-	elNode.FileIndex = 2
-	sample3 := nodeToMap(elNode)
-
-	err = printer.PrintResults(sample1)
-	if err != nil {
-		panic(err)
-	}
-
-	err = printer.PrintResults(sample2)
-	if err != nil {
-		panic(err)
-	}
-
-	err = printer.PrintResults(sample3)
-	if err != nil {
-		panic(err)
-	}
-
-	writer.Flush()
-	test.AssertResult(t, multiDocSample, output.String())
 }
 
 func TestPrinterMultipleDocsInSinglePrint(t *testing.T) {

release_instructions.txt

@@ -4,7 +4,6 @@
 - tag git with same version number
 - make sure local build passes
 - push tag to git
-  - 3.4.0, v3
   - git push --tags
 - make local xcompile (builds binaries for all platforms)
 
@@ -31,7 +30,7 @@
 
 - docker
   - build and push latest and new version tag
-  - docker build .  -t mikefarah/yq:latest -t mikefarah/yq:3 -t mikefarah/yq:3.X
+  - docker build .  -t mikefarah/yq:latest -t mikefarah/yq:VERSION 
 
 - debian package
   - ensure you get all vendor dependencies before packaging

scripts/check.sh

@@ -3,7 +3,7 @@
 set -o errexit
 set -o pipefail
 
-./bin/golangci-lint run --timeout=5m
+./bin/golangci-lint run
 
 # ./bin/golangci-lint \
 #   --tests \

scripts/format.sh

@@ -2,4 +2,3 @@
 
 find . \( -path ./vendor \) -prune -o -name "*.go" -exec goimports -w {} \;
 go mod tidy
-go mod vendor

snap/snapcraft.yaml

@@ -1,5 +1,5 @@
 name: yq
-version: '4.0.0'
+version: '4.0.0-beta2'
 summary: A lightweight and portable command-line YAML processor
 description: |
   The aim of the project is to be the jq or sed of yaml files.