1
0
mirror of https://github.com/taigrr/yq synced 2025-01-18 04:53:17 -08:00

Compare commits

..

14 Commits

Author SHA1 Message Date
Mike Farah
2937645fcd wip 2020-02-27 10:06:43 +11:00
Mike Farah
1a97b27041 wip - array length 2020-02-26 10:44:58 +11:00
Pascal Sochacki
da398765b8 added files for github action 2020-02-26 09:09:37 +11:00
Roberto Mier Escandon
d356fa0d0b Bump debian package to version 3.1-2
Updated all files to be more Debian compliant
Update release instructions for get mod vendor before releasing
2020-02-25 08:57:41 +11:00
Roberto Mier Escandon
d22bfc241b Add changelog 2020-02-25 08:57:41 +11:00
Mike Farah
954affea23 Create go.yml 2020-02-21 21:19:09 +11:00
Mike Farah
b0d1afb601 Update CONTRIBUTING.md 2020-02-21 21:16:47 +11:00
Mike Farah
b286636909 Create CODE_OF_CONDUCT.md 2020-02-21 21:13:14 +11:00
Mike Farah
bdf47c9797 Separated contribution notes 2020-02-21 21:12:09 +11:00
Mike Farah
1cc20d52bb Create CONTRIBUTING.md 2020-02-21 21:11:34 +11:00
Mike Farah
651d9edf88 Updating readme 2020-02-21 21:08:28 +11:00
Mike Farah
903605df39 Updating readme 2020-02-21 21:07:59 +11:00
Mike Farah
0f9facc84b Update issue templates 2020-02-21 21:04:54 +11:00
Mike Farah
5af86b1333 Update issue templates 2020-02-21 21:00:36 +11:00
23 changed files with 461 additions and 109 deletions

45
.github/ISSUE_TEMPLATE/bug_report.md vendored Normal file
View File

@@ -0,0 +1,45 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: bug
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
**Input Yaml**
Concise yaml document(s) (as simple as possible to show the bug)
data1.yml:
```yaml
this: should really work
```
data2.yml:
```yaml
but: it strangely didn't
```
**Command**
The command you ran:
```
yq merge data1.yml data2.yml
```
**Actual behavior**
```yaml
cat: meow
```
**Expected behavior**
```yaml
this: should really work
but: it strangely didn't
```
**Additional context**
Add any other context about the problem here.

View File

@@ -0,0 +1,36 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: enhancement
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
If we have data1.yml like:
```yaml
country: Australia
```
And we run a command:
```bash
yq predictWeather data1.yml
```
it could output
```yaml
temp: 32
```
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.

28
.github/workflows/go.yml vendored Normal file
View File

@@ -0,0 +1,28 @@
name: Go
on: [push]
jobs:
build:
name: Build
runs-on: ubuntu-latest
steps:
- name: Set up Go 1.13
uses: actions/setup-go@v1
with:
go-version: 1.13
id: go
- name: Check out code into the Go module directory
uses: actions/checkout@v2
- name: Get dependencies
run: |
go get -v -t -d ./...
if [ -f Gopkg.toml ]; then
curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
dep ensure
fi
- name: Build
run: make local test

76
CODE_OF_CONDUCT.md Normal file
View File

@@ -0,0 +1,76 @@
# Contributor Covenant Code of Conduct
## Our Pledge
In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, sex characteristics, gender identity and expression,
level of experience, education, socio-economic status, nationality, personal
appearance, race, religion, or sexual identity and orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment
include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or
advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.
## Scope
This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at mikefarah@gmail.com. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
[homepage]: https://www.contributor-covenant.org
For answers to common questions about this code of conduct, see
https://www.contributor-covenant.org/faq

8
CONTRIBUTING.md Normal file
View File

@@ -0,0 +1,8 @@
1. Install (golang)[https://golang.org/]
1. Run `scripts/devtools.sh` to install the required devtools
2. Run `make [local] vendor` to install the vendor dependencies
2. Run `make [local] test` to ensure you can run the existing tests
3. Write unit tests - (see existing examples). Changes will not be accepted without corresponding unit tests.
4. Make the code changes.
5. `make [local] test` to lint code and run tests
6. Profit! ok no profit, but raise a PR and get kudos :)

View File

@@ -12,15 +12,13 @@ V3 is officially out - if you've been using v2 and want/need to upgrade, checkou
## Install
### Download the latest binary
### [Download the latest binary](https://github.com/mikefarah/yq/releases/latest)
[Here](https://github.com/mikefarah/yq/releases/latest)
### On MacOS:
### MacOS:
```
brew install yq
```
### On Ubuntu and other Linux distros supporting `snap` packages:
### Ubuntu and other Linux distros supporting `snap` packages:
```
snap install yq
```
@@ -50,6 +48,8 @@ sudo add-apt-repository ppa:rmescandon/yq
sudo apt update
sudo apt install yq -y
```
Supported by @rmescandon
### Go Get:
```
GO111MODULE=on go get github.com/mikefarah/yq/v3
@@ -123,12 +123,3 @@ Flags:
Use "yq [command] --help" for more information about a command.
```
## Contribute
1. `scripts/devtools.sh`
2. `make [local] vendor`
3. add unit tests
4. apply changes to go.mod
5. `make [local] build`
7. profit

View File

@@ -1,74 +0,0 @@
Major release! Upgraded underlying yaml parser, re-written majority of yq. This has brought on a number of features that have been in demand for a while (see below).
This is in beta and needs some community feedback and testing :)
# New Features
- Keeps yaml comments and formatting, can specify yaml tags when updating. https://github.com/mikefarah/yq/issues/19, https://github.com/mikefarah/yq/issues/169, https://github.com/mikefarah/yq/issues/107, https://github.com/mikefarah/yq/issues/171, https://github.com/mikefarah/yq/issues/245, https://github.com/mikefarah/yq/issues/303,https://github.com/mikefarah/yq/issues/308,https://github.com/mikefarah/yq/issues/314
- Handles anchors! https://github.com/mikefarah/yq/issues/310, https://github.com/mikefarah/yq/issues/178
- Can print out matching paths and values when splatting https://github.com/mikefarah/yq/issues/20
- JSON output works for all commands! Yaml files with multiple documents are printed out as one JSON document per line.
- Deep splat (**) to match arbitrary paths
# Breaking changes
## Update scripts file format has changed to be more powerful.
Comments can be added, and delete commands have been introduced.
Before:
```yaml
b.e[+].name: Mike Farah
```
After:
```yaml
- command: update
path: b.e[+].thing
value:
#great
things: frog # wow!
- command: delete
path: b.d
```
https://github.com/mikefarah/yq/issues/305
## Reading and splatting, matching results are printed once per line.
e.g:
```json
parent:
childA:
no: matches here
childB:
there: matches
hi: no match
there2: also matches
```
```bash
yq r sample.yaml 'parent.*.there*'
```
old
```yaml
- null
- - matches
- also matches
```
new
```yaml
matches
also matches
```
and you can print the matching paths:
yq r --printMode pv sample.yaml 'parent.*.there*'
```yaml
parent.childB.there: matches
parent.childB.there2: also matches
```

11
action.yml Normal file
View File

@@ -0,0 +1,11 @@
name: 'YAML processor'
description: 'YAML processor for running in Github action'
inputs:
cmd:
description: 'The Command which should be run'
required: true
runs:
using: 'docker'
image: 'github-action/Dockerfile'
args:
- ${{ inputs.cmd }}

View File

@@ -190,6 +190,157 @@ func TestReadArrayCmd(t *testing.T) {
test.AssertResult(t, "b.e.[1].name: sam\n", result.Output)
}
func TestReadArrayLengthCmd(t *testing.T) {
content := `- things
- whatever
`
filename := test.WriteTempYamlFile(content)
defer test.RemoveTempYamlFile(filename)
cmd := getRootCommand()
result := test.RunCmd(cmd, fmt.Sprintf("read -l %s", filename))
if result.Error != nil {
t.Error(result.Error)
}
test.AssertResult(t, "2\n", result.Output)
}
func TestReadArrayLengthDeepCmd(t *testing.T) {
content := `holder:
- things
- whatever
`
filename := test.WriteTempYamlFile(content)
defer test.RemoveTempYamlFile(filename)
cmd := getRootCommand()
result := test.RunCmd(cmd, fmt.Sprintf("read -l %s holder", filename))
if result.Error != nil {
t.Error(result.Error)
}
test.AssertResult(t, "2\n", result.Output)
}
func TestReadArrayLengthDeepMultipleCmd(t *testing.T) {
content := `holderA:
- things
- whatever
holderB:
- other things
- cool
`
filename := test.WriteTempYamlFile(content)
defer test.RemoveTempYamlFile(filename)
cmd := getRootCommand()
result := test.RunCmd(cmd, fmt.Sprintf("read -l %s holder*", filename))
if result.Error != nil {
t.Error(result.Error)
}
test.AssertResult(t, "4\n", result.Output)
}
func TestReadArrayLengthDeepMultipleWithPathCmd(t *testing.T) {
content := `holderA:
- things
- whatever
holderB:
- other things
- cool
`
filename := test.WriteTempYamlFile(content)
defer test.RemoveTempYamlFile(filename)
cmd := getRootCommand()
result := test.RunCmd(cmd, fmt.Sprintf("read -l %s -ppv holder*", filename))
if result.Error != nil {
t.Error(result.Error)
}
test.AssertResult(t, "holderA: 2\nholderB: 2", result.Output)
}
func TestReadObjectLengthCmd(t *testing.T) {
content := `cat: meow
dog: bark
`
filename := test.WriteTempYamlFile(content)
defer test.RemoveTempYamlFile(filename)
cmd := getRootCommand()
result := test.RunCmd(cmd, fmt.Sprintf("read -l %s", filename))
if result.Error != nil {
t.Error(result.Error)
}
test.AssertResult(t, "2\n", result.Output)
}
func TestReadObjectLengthDeepCmd(t *testing.T) {
content := `holder:
cat: meow
dog: bark
`
filename := test.WriteTempYamlFile(content)
defer test.RemoveTempYamlFile(filename)
cmd := getRootCommand()
result := test.RunCmd(cmd, fmt.Sprintf("read -l %s holder", filename))
if result.Error != nil {
t.Error(result.Error)
}
test.AssertResult(t, "2\n", result.Output)
}
func TestReadObjectLengthDeepMultipleCmd(t *testing.T) {
content := `holderA:
cat: meow
dog: bark
holderB:
elephant: meow
zebra: bark
`
filename := test.WriteTempYamlFile(content)
defer test.RemoveTempYamlFile(filename)
cmd := getRootCommand()
result := test.RunCmd(cmd, fmt.Sprintf("read -l %s holder*", filename))
if result.Error != nil {
t.Error(result.Error)
}
test.AssertResult(t, "4\n", result.Output)
}
func TestReadObjectLengthDeepMultipleWithPathsCmd(t *testing.T) {
content := `holderA:
cat: meow
dog: bark
holderB:
elephant: meow
zebra: bark
`
filename := test.WriteTempYamlFile(content)
defer test.RemoveTempYamlFile(filename)
cmd := getRootCommand()
result := test.RunCmd(cmd, fmt.Sprintf("read -l -ppv %s holder*", filename))
if result.Error != nil {
t.Error(result.Error)
}
test.AssertResult(t, "holderA: 2\nholderB: 2\n", result.Output)
}
func TestReadScalarLengthCmd(t *testing.T) {
content := `meow`
filename := test.WriteTempYamlFile(content)
defer test.RemoveTempYamlFile(filename)
cmd := getRootCommand()
result := test.RunCmd(cmd, fmt.Sprintf("read -l %s", filename))
if result.Error != nil {
t.Error(result.Error)
}
test.AssertResult(t, "2\n", result.Output)
}
func TestReadDeepSplatCmd(t *testing.T) {
cmd := getRootCommand()
result := test.RunCmd(cmd, "read -p pv ../examples/sample.yaml b.**")

View File

@@ -7,6 +7,8 @@ import (
var customTag = ""
var printMode = "v"
var resultsAsArray = false
var printLength = false
var writeInplace = false
var writeScript = ""
var sourceYamlFile = ""

View File

@@ -26,6 +26,8 @@ yq r -- things.yaml '--key-starting-with-dashes.blah'
cmdRead.PersistentFlags().StringVarP(&docIndex, "doc", "d", "0", "process document index number (0 based, * for all documents)")
cmdRead.PersistentFlags().StringVarP(&printMode, "printMode", "p", "v", "print mode (v (values, default), p (paths), pv (path and value pairs)")
cmdRead.PersistentFlags().StringVarP(&defaultValue, "defaultValue", "D", "", "default value printed when there are no results")
cmdRead.PersistentFlags().BoolVarP(&resultsAsArray, "asArray", "a", false, "print results as array")
cmdRead.PersistentFlags().BoolVarP(&printLength, "length", "l", false, "print length of results")
cmdRead.PersistentFlags().BoolVarP(&explodeAnchors, "explodeAnchors", "X", false, "explode anchors")
return cmdRead
}

View File

@@ -77,7 +77,27 @@ func appendDocument(originalMatchingNodes []*yqlib.NodeContext, dataBucket yaml.
return append(originalMatchingNodes, matchingNodes...), nil
}
func printValue(node *yaml.Node, writer io.Writer) error {
func lengthOf(node *yaml.Node) int {
kindToCheck := node.Kind
if node.Kind == yaml.DocumentNode && len(node.Content) == 1 {
log.Debugf("length of document node, calculating length of child")
kindToCheck = node.Content[0].Kind
}
switch kindToCheck {
case yaml.ScalarNode:
return len(node.Value)
case yaml.MappingNode:
return len(node.Content) / 2
default:
return len(node.Content)
}
}
func printValue(node *yaml.Node, writer io.Writer, shouldPrintLength bool) error {
if shouldPrintLength {
return writeString(writer, fmt.Sprintf("%v\n", lengthOf(node)))
}
if node.Kind == yaml.ScalarNode {
_, errorWriting := writer.Write([]byte(node.Value + "\n"))
return errorWriting
@@ -159,6 +179,9 @@ func printResults(matchingNodes []*yqlib.NodeContext, writer io.Writer) error {
}
return nil
}
var counter = 0
var errorWriting error
for _, mappedDoc := range matchingNodes {
switch printMode {
@@ -172,17 +195,29 @@ func printResults(matchingNodes []*yqlib.NodeContext, writer io.Writer) error {
var parentNode = yaml.Node{Kind: yaml.MappingNode}
parentNode.Content = make([]*yaml.Node, 2)
parentNode.Content[0] = &yaml.Node{Kind: yaml.ScalarNode, Value: lib.PathStackToString(mappedDoc.PathStack)}
parentNode.Content[1] = mappedDoc.Node
if err := printValue(&parentNode, bufferedWriter); err != nil {
if printLength {
parentNode.Content[1] = &yaml.Node{Kind: yaml.ScalarNode, Value: fmt.Sprintf("%v", lengthOf(mappedDoc.Node))}
} else {
parentNode.Content[1] = mappedDoc.Node
}
if err := printValue(&parentNode, bufferedWriter, false); err != nil {
return err
}
default:
if err := printValue(mappedDoc.Node, bufferedWriter); err != nil {
if printLength {
counter = counter + lengthOf(mappedDoc.Node)
} else if err := printValue(mappedDoc.Node, bufferedWriter, false); err != nil {
return err
}
}
}
if printLength {
if err := writeString(bufferedWriter, fmt.Sprintf("%v\n", counter)); err != nil {
return err
}
}
return nil
}

23
debian/changelog vendored
View File

@@ -1,9 +1,24 @@
yq (3.1.1) eoan; urgency=medium
yq (3.1-2) eoan; urgency=medium
* Bug fix: yq 3 was removing empty inline-style objects and arrays (#355)
* Bug fix: Merge option returned different output when switching order of
merging files(#347)
* Bug fix: Add new object to existing array object was failing in 3.1.1 (#361)
* Bug fix: yq 3 empty keys did not allow merging of values (#356)
* Bug fix: keys quoted during merge (#363)
* Bug fix: Correct length with wc -l (#362)
* Bug fix: Write to empty document removed path (#359)
-- Roberto Mier Escandon <rmescandon@gmail.com> Mon, 24 Feb 2020 20:31:58 +0100
yq (3.1-1) eoan; urgency=medium
* Keeps yaml comments and formatting, can specify yaml tags when updating.
* Handles anchors! https://github.com/mikefarah/yq/issues/310, https://github.com/mikefarah/yq/issues/178
* Can print out matching paths and values when splatting https://github.com/mikefarah/yq/issues/20
* JSON output works for all commands! Yaml files with multiple documents are printed out as one JSON document per line.
* Handles anchors
* Can print out matching paths and values when splatting
* JSON output works for all commands
* Yaml files with multiple documents are printed out as one JSON
document per line.
* Deep splat (**) to match arbitrary paths
* Update scripts file format has changed to be more powerful
* Reading and splatting, matching results are printed once per line

2
debian/compat vendored
View File

@@ -1 +1 @@
9
10

5
debian/control vendored
View File

@@ -5,9 +5,8 @@ Maintainer: Roberto Mier EscandĂłn <rmescandon@gmail.com>
Build-Depends: debhelper (>=10),
dh-golang (>=1.34),
golang-1.13,
dpkg-dev,
rsync
Standards-Version: 3.9.6
Standards-Version: 4.1.4
Homepage: https://github.com/mikefarah/yq.git
Vcs-Browser: https://github.com/mikefarah/yq.git
Vcs-Git: https://github.com/mikefarah/yq.git
@@ -17,7 +16,7 @@ XSBC-Original-Maintainer: Roberto Mier EscandĂłn <rmescandon@gmail.com>
Package: yq
Architecture: any
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: a lightweight and portable command-line YAML processor
Description: lightweight and portable command-line YAML processor
.
The aim of the project is to be the
[jq](https://github.com/stedolan/jq) or sed of yaml files.

21
debian/copyright vendored
View File

@@ -3,5 +3,22 @@ Upstream-Name: yq
Source: https://github.com/mikefarah/yq.git
Files: *
Copyright: 2017 Mike Farah Ltd. All rights reserved
License: Proprietary
Copyright: 2017 Mike Farah
License: Expat
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
.
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

1
debian/files vendored Normal file
View File

@@ -0,0 +1 @@
yq_3.1-2_source.buildinfo devel optional

4
debian/rules vendored
View File

@@ -40,7 +40,7 @@ override_dh_auto_build:
mkdir -p ${SRCDIR}
mkdir -p ${GOBIN}
# copy project to local srcdir to build from there
rsync -avz --progress --exclude=_build --exclude=debian --exclude=tmp. --exclude=go.mod . $(SRCDIR)
rsync -avz --progress --exclude=_build --exclude=debian --exclude=tmp. --exclude=go.mod --exclude=docs . $(SRCDIR)
# build go code
(cd ${SRCDIR} && go install -buildmode=pie ./...)
@@ -48,8 +48,6 @@ override_dh_auto_test:
(cd ${SRCDIR} && go test -v ./...)
override_dh_auto_install:
mkdir -p ${DESTDIR}/${BINDIR}
mkdir -p ${DESTDIR}/${ASSETSDIR}
cp ${GOBIN}/yq ${DESTDIR}/${BINDIR}
cp -f ${SRCDIR}/LICENSE ${DESTDIR}/${ASSETSDIR}
chmod a+x ${DESTDIR}/${BINDIR}/yq

3
debian/yq.dirs vendored Normal file
View File

@@ -0,0 +1,3 @@
usr/bin
usr/share/yq
usr/share/man/man1

5
github-action/Dockerfile Normal file
View File

@@ -0,0 +1,5 @@
FROM mikefarah/yq:3
COPY entrypoint.sh /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

4
github-action/entrypoint.sh Executable file
View File

@@ -0,0 +1,4 @@
#!/bin/sh -l
echo "$1"
eval $1

View File

@@ -33,6 +33,8 @@
- docker build . -t mikefarah/yq:latest -t mikefarah/yq:VERSION
- debian package
- ensure you get all vendor dependencies before packaging
```go mod vendor```
- execute
```dch -i```
- fill debian/changelog with changes from last version
@@ -42,4 +44,4 @@
- put to PPA
```dput ppa:<REPOSITORY> ../yq_<VERSION>_source.changes```
(current distro repository is ppa:rmescandon/yq. In case that a new version
is released, please contact rmescandon@gmail.com to bump debian package)
is released, please contact rmescandon@gmail.com to bump debian package)

View File

@@ -1,3 +0,0 @@
b:
c: thing
d: another thing