taigrr/arc
1
0
Fork 0
mirror of https://github.com/taigrr/arc synced 2025-01-18 04:33:13 -08:00
Code Issues Projects Releases Wiki Activity

initial import

Browse Source
This commit is contained in:
Will
2016-05-22 22:16:37 +09:00
commit 0075ef607f
256 changed files with 58234 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
vendor/github.com/klauspost/compress/LICENSE generated vendored Normal file
View File

@@ -0,0 +1,27 @@
Copyright (c) 2012 The Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

330
vendor/github.com/klauspost/compress/README.md generated vendored Normal file
View File

@@ -0,0 +1,330 @@
# compress
This package is based on an optimized Deflate function, which is used by gzip/zip/zlib packages.
It offers slightly better compression at lower compression settings, and up to 3x faster encoding at highest compression level.
* [High Throughput Benchmark](http://blog.klauspost.com/go-gzipdeflate-benchmarks/).
* [Small Payload/Webserver Benchmarks](http://blog.klauspost.com/gzip-performance-for-go-webservers/).
* [Linear Time Compression](http://blog.klauspost.com/constant-time-gzipzip-compression/).
* [Re-balancing Deflate Compression Levels](https://blog.klauspost.com/rebalancing-deflate-compression-levels/)
[![Build Status](https://travis-ci.org/klauspost/compress.svg?branch=master)](https://travis-ci.org/klauspost/compress)
# changelog
* Mar 24, 2016: Always attempt Huffman encoding on level 4-7. This improves base 64 encoded data compression.
* Mar 24, 2016: Small speedup for level 1-3.
* Feb 19, 2016: Faster bit writer, level -2 is 15% faster, level 1 is 4% faster.
* Feb 19, 2016: Handle small payloads faster in level 1-3.
* Feb 19, 2016: Added faster level 2 + 3 compression modes.
* Feb 19, 2016: [Rebalanced compression levels](https://blog.klauspost.com/rebalancing-deflate-compression-levels/), so there is a more even progresssion in terms of compression. New default level is 5.
* Feb 14, 2016: Snappy: Merge upstream changes.
* Feb 14, 2016: Snappy: Fix aggressive skipping.
* Feb 14, 2016: Snappy: Update benchmark.
* Feb 13, 2016: Deflate: Fixed assembler problem that could lead to sub-optimal compression.
* Feb 12, 2016: Snappy: Added AMD64 SSE 4.2 optimizations to matching, which makes easy to compress material run faster. Typical speedup is around 25%.
* Feb 9, 2016: Added Snappy package fork. This version is 5-7% faster, much more on hard to compress content.
* Jan 30, 2016: Optimize level 1 to 3 by not considering static dictionary or storing uncompressed. ~4-5% speedup.
* Jan 16, 2016: Optimization on deflate level 1,2,3 compression.
* Jan 8 2016: Merge [CL 18317](https://go-review.googlesource.com/#/c/18317): fix reading, writing of zip64 archives.
* Dec 8 2015: Make level 1 and -2 deterministic even if write size differs.
* Dec 8 2015: Split encoding functions, so hashing and matching can potentially be inlined. 1-3% faster on AMD64. 5% faster on other platforms.
* Dec 8 2015: Fixed rare [one byte out-of bounds read](https://github.com/klauspost/compress/issues/20). Please update!
* Nov 23 2015: Optimization on token writer. ~2-4% faster. Contributed by [@dsnet](https://github.com/dsnet).
* Nov 20 2015: Small optimization to bit writer on 64 bit systems.
* Nov 17 2015: Fixed out-of-bound errors if the underlying Writer returned an error. See [#15](https://github.com/klauspost/compress/issues/15).
* Nov 12 2015: Added [io.WriterTo](https://golang.org/pkg/io/#WriterTo) support to gzip/inflate.
* Nov 11 2015: Merged [CL 16669](https://go-review.googlesource.com/#/c/16669/4): archive/zip: enable overriding (de)compressors per file
* Oct 15 2015: Added skipping on uncompressible data. Random data speed up >5x.
# usage
The packages are drop-in replacements for standard libraries. Simply replace the import path to use them:
| old import | new import |
|--------------------|-----------------------------------------|
| `compress/gzip` | `github.com/klauspost/compress/gzip` |
| `compress/zlib` | `github.com/klauspost/compress/zlib` |
| `archive/zip` | `github.com/klauspost/compress/zip` |
| `compress/deflate` | `github.com/klauspost/compress/deflate` |
| `github.com/golang/snappy` | `github.com/klauspost/compress/snappy` |
You may also be interested in [pgzip](https://github.com/klauspost/pgzip), which is a drop in replacement for gzip, which support multithreaded compression on big files and the optimized [crc32](https://github.com/klauspost/crc32) package used by these packages.
The packages contains the same as the standard library, so you can use the godoc for that: [gzip](http://golang.org/pkg/compress/gzip/), [zip](http://golang.org/pkg/archive/zip/), [zlib](http://golang.org/pkg/compress/zlib/), [flate](http://golang.org/pkg/compress/flate/), [snappy](http://golang.org/pkg/compress/snappy/).
Currently there is only minor speedup on decompression (mostly CRC32 calculation).
# deflate optimizations
* Minimum matches are 4 bytes, this leads to fewer searches and better compression.
* Stronger hash (iSCSI CRC32) for matches on x64 with SSE 4.2 support. This leads to fewer hash collisions.
* Literal byte matching using SSE 4.2 for faster match comparisons.
* Bulk hashing on matches.
* Much faster dictionary indexing with `NewWriterDict()`/`Reset()`.
* Make Bit Coder faster by assuming we are on a 64 bit CPU.
* Level 1 compression replaced by converted "Snappy" algorithm.
* Uncompressible content is detected and skipped faster.
* A lot of branching eliminated by having two encoders for levels 2+3 and 4+.
* All heap memory allocations eliminated.
```
benchmark old ns/op new ns/op delta
BenchmarkEncodeDigitsSpeed1e4-4 554029 265175 -52.14%
BenchmarkEncodeDigitsSpeed1e5-4 3908558 2416595 -38.17%
BenchmarkEncodeDigitsSpeed1e6-4 37546692 24875330 -33.75%
BenchmarkEncodeDigitsDefault1e4-4 781510 486322 -37.77%
BenchmarkEncodeDigitsDefault1e5-4 15530248 6740175 -56.60%
BenchmarkEncodeDigitsDefault1e6-4 174915710 76498625 -56.27%
BenchmarkEncodeDigitsCompress1e4-4 769995 485652 -36.93%
BenchmarkEncodeDigitsCompress1e5-4 15450113 6929589 -55.15%
BenchmarkEncodeDigitsCompress1e6-4 175114660 73348495 -58.11%
BenchmarkEncodeTwainSpeed1e4-4 560122 275977 -50.73%
BenchmarkEncodeTwainSpeed1e5-4 3740978 2506095 -33.01%
BenchmarkEncodeTwainSpeed1e6-4 35542802 21904440 -38.37%
BenchmarkEncodeTwainDefault1e4-4 828534 549026 -33.74%
BenchmarkEncodeTwainDefault1e5-4 13667153 7528455 -44.92%
BenchmarkEncodeTwainDefault1e6-4 141191770 79952170 -43.37%
BenchmarkEncodeTwainCompress1e4-4 830050 545694 -34.26%
BenchmarkEncodeTwainCompress1e5-4 16620852 8460600 -49.10%
BenchmarkEncodeTwainCompress1e6-4 193326820 90808750 -53.03%
benchmark old MB/s new MB/s speedup
BenchmarkEncodeDigitsSpeed1e4-4 18.05 37.71 2.09x
BenchmarkEncodeDigitsSpeed1e5-4 25.58 41.38 1.62x
BenchmarkEncodeDigitsSpeed1e6-4 26.63 40.20 1.51x
BenchmarkEncodeDigitsDefault1e4-4 12.80 20.56 1.61x
BenchmarkEncodeDigitsDefault1e5-4 6.44 14.84 2.30x
BenchmarkEncodeDigitsDefault1e6-4 5.72 13.07 2.28x
BenchmarkEncodeDigitsCompress1e4-4 12.99 20.59 1.59x
BenchmarkEncodeDigitsCompress1e5-4 6.47 14.43 2.23x
BenchmarkEncodeDigitsCompress1e6-4 5.71 13.63 2.39x
BenchmarkEncodeTwainSpeed1e4-4 17.85 36.23 2.03x
BenchmarkEncodeTwainSpeed1e5-4 26.73 39.90 1.49x
BenchmarkEncodeTwainSpeed1e6-4 28.14 45.65 1.62x
BenchmarkEncodeTwainDefault1e4-4 12.07 18.21 1.51x
BenchmarkEncodeTwainDefault1e5-4 7.32 13.28 1.81x
BenchmarkEncodeTwainDefault1e6-4 7.08 12.51 1.77x
BenchmarkEncodeTwainCompress1e4-4 12.05 18.33 1.52x
BenchmarkEncodeTwainCompress1e5-4 6.02 11.82 1.96x
BenchmarkEncodeTwainCompress1e6-4 5.17 11.01 2.13x
```
* "Speed" is compression level 1
* "Default" is compression level 6
* "Compress" is compression level 9
* Test files are [Digits](https://github.com/klauspost/compress/blob/master/testdata/e.txt) (no matches) and [Twain](https://github.com/klauspost/compress/blob/master/testdata/Mark.Twain-Tom.Sawyer.txt) (plain text) .
As can be seen it shows a very good speedup all across the line.
`Twain` is a much more realistic benchmark, and will be closer to JSON/HTML performance. Here speed is equivalent or faster, up to 2 times.
**Without assembly**. This is what you can expect on systems that does not have amd64 and SSE 4:
```
benchmark old ns/op new ns/op delta
BenchmarkEncodeDigitsSpeed1e4-4 554029 249558 -54.96%
BenchmarkEncodeDigitsSpeed1e5-4 3908558 2295216 -41.28%
BenchmarkEncodeDigitsSpeed1e6-4 37546692 22594905 -39.82%
BenchmarkEncodeDigitsDefault1e4-4 781510 579850 -25.80%
BenchmarkEncodeDigitsDefault1e5-4 15530248 10096561 -34.99%
BenchmarkEncodeDigitsDefault1e6-4 174915710 111470780 -36.27%
BenchmarkEncodeDigitsCompress1e4-4 769995 579708 -24.71%
BenchmarkEncodeDigitsCompress1e5-4 15450113 10266373 -33.55%
BenchmarkEncodeDigitsCompress1e6-4 175114660 110170120 -37.09%
BenchmarkEncodeTwainSpeed1e4-4 560122 260679 -53.46%
BenchmarkEncodeTwainSpeed1e5-4 3740978 2097372 -43.94%
BenchmarkEncodeTwainSpeed1e6-4 35542802 20353449 -42.74%
BenchmarkEncodeTwainDefault1e4-4 828534 646016 -22.03%
BenchmarkEncodeTwainDefault1e5-4 13667153 10056369 -26.42%
BenchmarkEncodeTwainDefault1e6-4 141191770 105268770 -25.44%
BenchmarkEncodeTwainCompress1e4-4 830050 642401 -22.61%
BenchmarkEncodeTwainCompress1e5-4 16620852 11157081 -32.87%
BenchmarkEncodeTwainCompress1e6-4 193326820 121780770 -37.01%
benchmark old MB/s new MB/s speedup
BenchmarkEncodeDigitsSpeed1e4-4 18.05 40.07 2.22x
BenchmarkEncodeDigitsSpeed1e5-4 25.58 43.57 1.70x
BenchmarkEncodeDigitsSpeed1e6-4 26.63 44.26 1.66x
BenchmarkEncodeDigitsDefault1e4-4 12.80 17.25 1.35x
BenchmarkEncodeDigitsDefault1e5-4 6.44 9.90 1.54x
BenchmarkEncodeDigitsDefault1e6-4 5.72 8.97 1.57x
BenchmarkEncodeDigitsCompress1e4-4 12.99 17.25 1.33x
BenchmarkEncodeDigitsCompress1e5-4 6.47 9.74 1.51x
BenchmarkEncodeDigitsCompress1e6-4 5.71 9.08 1.59x
BenchmarkEncodeTwainSpeed1e4-4 17.85 38.36 2.15x
BenchmarkEncodeTwainSpeed1e5-4 26.73 47.68 1.78x
BenchmarkEncodeTwainSpeed1e6-4 28.14 49.13 1.75x
BenchmarkEncodeTwainDefault1e4-4 12.07 15.48 1.28x
BenchmarkEncodeTwainDefault1e5-4 7.32 9.94 1.36x
BenchmarkEncodeTwainDefault1e6-4 7.08 9.50 1.34x
BenchmarkEncodeTwainCompress1e4-4 12.05 15.57 1.29x
BenchmarkEncodeTwainCompress1e5-4 6.02 8.96 1.49x
BenchmarkEncodeTwainCompress1e6-4 5.17 8.21 1.59x
```
So even without the assembly optimizations there is a general speedup across the board.
## level 1-3 "snappy" compression
Level 1 "Best Speed" is completely replaced by a converted version of the algorithm found in Snappy, modified to be fully
compatible with the deflate bitstream (and thus still compatible with all existing zlib/gzip libraries and tools).
This version is considerably faster than the "old" deflate at level 1. It does however come at a compression loss, usually in the order of 3-4% compared to the old level 1. However, the speed is usually 1.75 times that of the fastest deflate mode.
In my previous experiments the most common case for "level 1" was that it provided no significant speedup, only lower compression compared to level 2 and sometimes even 3. However, the modified Snappy algorithm provides a very good sweet spot. Usually about 75% faster and with only little compression loss. Therefore I decided to *replace* level 1 with this mode entirely.
Input is split into blocks of 64kb of, and they are encoded independently (no backreferences across blocks) for the best speed. Contrary to Snappy the output is entropy-encoded, so you will almost always see better compression than Snappy. But Snappy is still about twice as fast as Snappy in deflate mode.
Level 2 and 3 have also been replaced. Level 2 is capable is matching between blocks and level 3 checks up to two hashes for matches before choosing the longest for encoding the match.
## compression levels
This table shows the compression at each level, and the percentage of the output size compared to output
at the similar level with the standard library. Compression data is `Twain`, see above.
(Not up-to-date after rebalancing)
| Level | Bytes | % size |
|-------|--------|--------|
| 1 | 194622 | 103.7% |
| 2 | 174684 | 96.85% |
| 3 | 170301 | 98.45% |
| 4 | 165253 | 97.69% |
| 5 | 161274 | 98.65% |
| 6 | 160464 | 99.71% |
| 7 | 160304 | 99.87% |
| 8 | 160279 | 99.99% |
| 9 | 160279 | 99.99% |
To interpret and example, this version of deflate compresses input of 407287 bytes to 161274 bytes at level 5, which is 98.6% of the size of what the standard library produces; 161274 bytes.
This means that from level 4 you can expect a compression level increase of a few percent. Level 1 is about 3% worse, as descibed above.
# linear time compression
This compression library adds a special compression level, named `ConstantCompression`, which allows near linear time compression. This is done by completely disabling matching of previous data, and only reduce the number of bits to represent each character.
This means that often used characters, like 'e' and ' ' (space) in text use the fewest bits to represent, and rare characters like '¤' takes more bits to represent. For more information see [wikipedia](https://en.wikipedia.org/wiki/Huffman_coding) or this nice [video](https://youtu.be/ZdooBTdW5bM).
Since this type of compression has much less variance, the compression speed is mostly unaffected by the input data, and is usually more than *180MB/s* for a single core.
The downside is that the compression ratio is usually considerably worse than even the fastest conventional compression. The compression raio can never be better than 8:1 (12.5%).
The linear time compression can be used as a "better than nothing" mode, where you cannot risk the encoder to slow down on some content. For comparison, the size of the "Twain" text is *233460 bytes* (+29% vs. level 1) and encode speed is 144MB/s (4.5x level 1). So in this case you trade a 30% size increase for a 4 times speedup.
For more information see my blog post on [Fast Linear Time Compression](http://blog.klauspost.com/constant-time-gzipzip-compression/).
# gzip/zip optimizations
* Uses the faster deflate
* Uses SSE 4.2 CRC32 calculations.
Speed increase is up to 3x of the standard library, but usually around 2x.
This is close to a real world benchmark as you will get. A 2.3MB JSON file. (NOTE: not up-to-date)
```
benchmark old ns/op new ns/op delta
BenchmarkGzipL1-4 95212470 59938275 -37.05%
BenchmarkGzipL2-4 102069730 76349195 -25.20%
BenchmarkGzipL3-4 115472770 82492215 -28.56%
BenchmarkGzipL4-4 153197780 107570890 -29.78%
BenchmarkGzipL5-4 203930260 134387930 -34.10%
BenchmarkGzipL6-4 233172100 145495400 -37.60%
BenchmarkGzipL7-4 297190260 197926950 -33.40%
BenchmarkGzipL8-4 512819750 376244733 -26.63%
BenchmarkGzipL9-4 563366800 403266833 -28.42%
benchmark old MB/s new MB/s speedup
BenchmarkGzipL1-4 52.11 82.78 1.59x
BenchmarkGzipL2-4 48.61 64.99 1.34x
BenchmarkGzipL3-4 42.97 60.15 1.40x
BenchmarkGzipL4-4 32.39 46.13 1.42x
BenchmarkGzipL5-4 24.33 36.92 1.52x
BenchmarkGzipL6-4 21.28 34.10 1.60x
BenchmarkGzipL7-4 16.70 25.07 1.50x
BenchmarkGzipL8-4 9.68 13.19 1.36x
BenchmarkGzipL9-4 8.81 12.30 1.40x
```
Multithreaded compression using [pgzip](https://github.com/klauspost/pgzip) comparison, Quadcore, CPU = 8:
(Not updated, old numbers)
```
benchmark old ns/op new ns/op delta
BenchmarkGzipL1 96155500 25981486 -72.98%
BenchmarkGzipL2 101905830 24601408 -75.86%
BenchmarkGzipL3 113506490 26321506 -76.81%
BenchmarkGzipL4 143708220 31761818 -77.90%
BenchmarkGzipL5 188210770 39602266 -78.96%
BenchmarkGzipL6 209812000 40402313 -80.74%
BenchmarkGzipL7 270015440 56103210 -79.22%
BenchmarkGzipL8 461359700 91255220 -80.22%
BenchmarkGzipL9 498361833 88755075 -82.19%
benchmark old MB/s new MB/s speedup
BenchmarkGzipL1 51.60 190.97 3.70x
BenchmarkGzipL2 48.69 201.69 4.14x
BenchmarkGzipL3 43.71 188.51 4.31x
BenchmarkGzipL4 34.53 156.22 4.52x
BenchmarkGzipL5 26.36 125.29 4.75x
BenchmarkGzipL6 23.65 122.81 5.19x
BenchmarkGzipL7 18.38 88.44 4.81x
BenchmarkGzipL8 10.75 54.37 5.06x
BenchmarkGzipL9 9.96 55.90 5.61x
```
# snappy package
### This is still in development, and should not be used for critical applications.
The Snappy package contains some optimizations over the standard package.
This speeds up mainly **hard** and **easy** to compress material.
Here are the "standard" benchmarks, compared to current Snappy master (13 feb 2016).
## Speed
```
name old speed new speed delta
WordsDecode1e3-8 405MB/s ± 5% 444MB/s ± 1% +9.60% (p=0.045 n=3+3)
WordsEncode1e1-8 4.55MB/s ± 1% 98.93MB/s ± 2% +2075.95% (p=0.000 n=3+3)
WordsEncode1e2-8 36.4MB/s ± 0% 166.1MB/s ± 3% +356.03% (p=0.000 n=3+3)
WordsEncode1e3-8 129MB/s ± 0% 185MB/s ± 1% +43.82% (p=0.000 n=3+3)
WordsEncode1e5-8 125MB/s ± 1% 140MB/s ± 2% +11.77% (p=0.005 n=3+3)
WordsEncode1e6-8 121MB/s ± 3% 134MB/s ± 0% +11.15% (p=0.026 n=3+3)
RandomEncode-8 2.80GB/s ± 2% 2.68GB/s ± 1% -4.32% (p=0.019 n=3+3)
_UFlat3-8 746MB/s ± 2% 812MB/s ± 1% +8.90% (p=0.004 n=3+3)
_UFlat4-8 2.50GB/s ± 1% 3.06GB/s ± 1% +22.68% (p=0.000 n=3+3)
_ZFlat0-8 284MB/s ± 1% 362MB/s ± 1% +27.45% (p=0.000 n=3+3)
_ZFlat2-8 2.85GB/s ± 0% 3.71GB/s ± 1% +30.21% (p=0.000 n=3+3)
_ZFlat3-8 64.5MB/s ± 1% 216.9MB/s ± 2% +236.02% (p=0.000 n=3+3)
_ZFlat4-8 415MB/s ± 1% 2000MB/s ± 1% +382.43% (p=0.000 n=3+3)
_ZFlat5-8 282MB/s ± 1% 354MB/s ± 2% +25.67% (p=0.003 n=3+3)
_ZFlat6-8 124MB/s ± 1% 136MB/s ± 2% +9.84% (p=0.013 n=3+3)
_ZFlat7-8 116MB/s ± 2% 127MB/s ± 1% +10.12% (p=0.002 n=3+3)
_ZFlat8-8 128MB/s ± 1% 142MB/s ± 1% +11.38% (p=0.000 n=3+3)
_ZFlat9-8 111MB/s ± 2% 120MB/s ± 1% +8.45% (p=0.009 n=3+3)
_ZFlat10-8 318MB/s ± 1% 439MB/s ± 1% +38.16% (p=0.000 n=3+3)
_ZFlat11-8 183MB/s ± 0% 226MB/s ± 3% +23.53% (p=0.004 n=3+3)
```
Only significant differences are included.
## Size Comparison:
```
name data insize outsize ref red. ref-red r-delta
Flat0: html 102400 23317 23330 77.23% 77.23% 0.01%
Flat1: urls 712086 337290 335282 52.63% 52.63% -0.28%
Flat2: jpg 123093 123035 123032 0.05% 0.05% -0.00%
Flat3: jpg_200 123093 123035 123032 0.05% 0.05% -0.00%
Flat4: pdf 102400 84897 83754 17.09% 17.09% -1.12%
Flat5: html4 409600 92689 92366 77.37% 77.37% -0.08%
Flat6: txt1 152089 89544 89495 41.12% 41.12% -0.03%
Flat7: txt2 129301 80531 80518 37.72% 37.72% -0.01%
Flat8: txt3 426754 238857 238849 44.03% 44.03% -0.00%
Flat9: txt4 481861 324755 325047 32.60% 32.60% 0.06%
Flat10: pb 118588 24723 23392 79.15% 79.15% -1.12%
Flat11: gaviota 184320 73963 73962 59.87% 59.87% -0.00%
```
r-delta is difference in compression. Negative means this package performs worse.
# license
This code is licensed under the same conditions as the original Go code. See LICENSE file.

193
vendor/github.com/klauspost/compress/flate/asm_test.go generated vendored Normal file
View File

@@ -0,0 +1,193 @@
// Copyright 2015, Klaus Post, see LICENSE for details.
//+build amd64
package flate
import (
"math/rand"
"testing"
)
func TestCRC(t *testing.T) {
if !useSSE42 {
t.Skip("Skipping CRC test, no SSE 4.2 available")
}
for _, x := range deflateTests {
y := x.out
if len(y) >= minMatchLength {
t.Logf("In: %v, Out:0x%08x", y[0:minMatchLength], crc32sse(y[0:minMatchLength]))
}
}
}
func TestCRCBulk(t *testing.T) {
if !useSSE42 {
t.Skip("Skipping CRC test, no SSE 4.2 available")
}
for _, x := range deflateTests {
y := x.out
y = append(y, y...)
y = append(y, y...)
y = append(y, y...)
y = append(y, y...)
y = append(y, y...)
y = append(y, y...)
if !testing.Short() {
y = append(y, y...)
y = append(y, y...)
}
y = append(y, 1)
if len(y) >= minMatchLength {
for j := len(y) - 1; j >= 4; j-- {
// Create copy, so we easier detect of-of-bound reads
test := make([]byte, j)
test2 := make([]byte, j)
copy(test, y[:j])
copy(test2, y[:j])
// We allocate one more than we need to test for unintentional overwrites
dst := make([]hash, j-3+1)
ref := make([]hash, j-3+1)
for i := range dst {
dst[i] = hash(i + 100)
ref[i] = hash(i + 101)
}
// Last entry must NOT be overwritten.
dst[j-3] = 0x1234
ref[j-3] = 0x1234
// Do two encodes we can compare
crc32sseAll(test, dst)
crc32sseAll(test2, ref)
// Check all values
for i, got := range dst {
if i == j-3 {
if dst[i] != 0x1234 {
t.Fatalf("end of expected dst overwritten, was %08x", uint32(dst[i]))
}
continue
}
expect := crc32sse(y[i : i+4])
if got != expect && got == hash(i)+100 {
t.Errorf("Len:%d Index:%d, expected 0x%08x but not modified", len(y), i, uint32(expect))
} else if got != expect {
t.Errorf("Len:%d Index:%d, got 0x%08x expected:0x%08x", len(y), i, uint32(got), uint32(expect))
}
expect = ref[i]
if got != expect {
t.Errorf("Len:%d Index:%d, got 0x%08x expected:0x%08x", len(y), i, got, expect)
}
}
}
}
}
}
func TestMatchLen(t *testing.T) {
if !useSSE42 {
t.Skip("Skipping Matchlen test, no SSE 4.2 available")
}
// Maximum length tested
var maxLen = 512
// Skips per iteration
is, js, ks := 3, 2, 1
if testing.Short() {
is, js, ks = 7, 5, 3
}
a := make([]byte, maxLen)
b := make([]byte, maxLen)
bb := make([]byte, maxLen)
rand.Seed(1)
for i := range a {
a[i] = byte(rand.Int63())
b[i] = byte(rand.Int63())
}
// Test different lengths
for i := 0; i < maxLen; i += is {
// Test different dst offsets.
for j := 0; j < maxLen-1; j += js {
copy(bb, b)
// Test different src offsets
for k := i - 1; k >= 0; k -= ks {
copy(bb[j:], a[k:i])
maxTest := maxLen - j
if maxTest > maxLen-k {
maxTest = maxLen - k
}
got := matchLenSSE4(a[k:], bb[j:], maxTest)
expect := matchLenReference(a[k:], bb[j:], maxTest)
if got > maxTest || got < 0 {
t.Fatalf("unexpected result %d (len:%d, src offset: %d, dst offset:%d)", got, maxTest, k, j)
}
if got != expect {
t.Fatalf("Mismatch, expected %d, got %d", expect, got)
}
}
}
}
}
// matchLenReference is a reference matcher.
func matchLenReference(a, b []byte, max int) int {
for i := 0; i < max; i++ {
if a[i] != b[i] {
return i
}
}
return max
}
func TestHistogram(t *testing.T) {
if !useSSE42 {
t.Skip("Skipping Matchlen test, no SSE 4.2 available")
}
// Maximum length tested
const maxLen = 65536
var maxOff = 8
// Skips per iteration
is, js := 5, 3
if testing.Short() {
is, js = 9, 1
maxOff = 1
}
a := make([]byte, maxLen+maxOff)
rand.Seed(1)
for i := range a {
a[i] = byte(rand.Int63())
}
// Test different lengths
for i := 0; i <= maxLen; i += is {
// Test different offsets
for j := 0; j < maxOff; j += js {
var got [256]int32
var reference [256]int32
histogram(a[j:i+j], got[:])
histogramReference(a[j:i+j], reference[:])
for k := range got {
if got[k] != reference[k] {
t.Fatalf("mismatch at len:%d, offset:%d, value %d: (got) %d != %d (expected)", i, j, k, got[k], reference[k])
}
}
}
}
}
// histogramReference is a reference
func histogramReference(b []byte, h []int32) {
if len(h) < 256 {
panic("Histogram too small")
}
for _, t := range b {
h[t]++
}
}

32
vendor/github.com/klauspost/compress/flate/copy.go generated vendored Normal file
View File

@@ -0,0 +1,32 @@
// Copyright 2012 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
// forwardCopy is like the built-in copy function except that it always goes
// forward from the start, even if the dst and src overlap.
// It is equivalent to:
// for i := 0; i < n; i++ {
// mem[dst+i] = mem[src+i]
// }
func forwardCopy(mem []byte, dst, src, n int) {
if dst <= src {
copy(mem[dst:dst+n], mem[src:src+n])
return
}
for {
if dst >= src+n {
copy(mem[dst:dst+n], mem[src:src+n])
return
}
// There is some forward overlap. The destination
// will be filled with a repeated pattern of mem[src:src+k].
// We copy one instance of the pattern here, then repeat.
// Each time around this loop k will double.
k := dst - src
copy(mem[dst:dst+k], mem[src:src+k])
n -= k
dst += k
}
}

54
vendor/github.com/klauspost/compress/flate/copy_test.go generated vendored Normal file
View File

@@ -0,0 +1,54 @@
// Copyright 2012 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
import (
"testing"
)
func TestForwardCopy(t *testing.T) {
testCases := []struct {
dst0, dst1 int
src0, src1 int
want string
}{
{0, 9, 0, 9, "012345678"},
{0, 5, 4, 9, "45678"},
{4, 9, 0, 5, "01230"},
{1, 6, 3, 8, "34567"},
{3, 8, 1, 6, "12121"},
{0, 9, 3, 6, "345"},
{3, 6, 0, 9, "012"},
{1, 6, 0, 9, "00000"},
{0, 4, 7, 8, "7"},
{0, 1, 6, 8, "6"},
{4, 4, 6, 9, ""},
{2, 8, 6, 6, ""},
{0, 0, 0, 0, ""},
}
for _, tc := range testCases {
b := []byte("0123456789")
n := tc.dst1 - tc.dst0
if tc.src1-tc.src0 < n {
n = tc.src1 - tc.src0
}
forwardCopy(b, tc.dst0, tc.src0, n)
got := string(b[tc.dst0 : tc.dst0+n])
if got != tc.want {
t.Errorf("dst=b[%d:%d], src=b[%d:%d]: got %q, want %q",
tc.dst0, tc.dst1, tc.src0, tc.src1, got, tc.want)
}
// Check that the bytes outside of dst[:n] were not modified.
for i, x := range b {
if i >= tc.dst0 && i < tc.dst0+n {
continue
}
if int(x) != '0'+i {
t.Errorf("dst=b[%d:%d], src=b[%d:%d]: copy overrun at b[%d]: got '%c', want '%c'",
tc.dst0, tc.dst1, tc.src0, tc.src1, i, x, '0'+i)
}
}
}
}

39
vendor/github.com/klauspost/compress/flate/crc32_amd64.go generated vendored Normal file
View File

@@ -0,0 +1,39 @@
//+build !noasm
//+build !appengine
// Copyright 2015, Klaus Post, see LICENSE for details.
package flate
import (
"github.com/klauspost/cpuid"
)
// crc32sse returns a hash for the first 4 bytes of the slice
// len(a) must be >= 4.
//go:noescape
func crc32sse(a []byte) hash
// crc32sseAll calculates hashes for each 4-byte set in a.
// dst must be east len(a) - 4 in size.
// The size is not checked by the assembly.
//go:noescape
func crc32sseAll(a []byte, dst []hash)
// matchLenSSE4 returns the number of matching bytes in a and b
// up to length 'max'. Both slices must be at least 'max'
// bytes in size.
// It uses the PCMPESTRI SSE 4.2 instruction.
//go:noescape
func matchLenSSE4(a, b []byte, max int) int
// histogram accumulates a histogram of b in h.
// h must be at least 256 entries in length,
// and must be cleared before calling this function.
//go:noescape
func histogram(b []byte, h []int32)
// Detect SSE 4.2 feature.
func init() {
useSSE42 = cpuid.CPU.SSE42()
}

219
vendor/github.com/klauspost/compress/flate/crc32_amd64.s generated vendored Normal file
View File

@@ -0,0 +1,219 @@
//+build !noasm
//+build !appengine
// Copyright 2015, Klaus Post, see LICENSE for details.
// func crc32sse(a []byte) hash
TEXT ·crc32sse(SB), 4, $0
MOVQ a+0(FP), R10
XORQ BX, BX
// CRC32 dword (R10), EBX
BYTE $0xF2; BYTE $0x41; BYTE $0x0f
BYTE $0x38; BYTE $0xf1; BYTE $0x1a
MOVL BX, ret+24(FP)
RET
// func crc32sseAll(a []byte, dst []hash)
TEXT ·crc32sseAll(SB), 4, $0
MOVQ a+0(FP), R8 // R8: src
MOVQ a_len+8(FP), R10 // input length
MOVQ dst+24(FP), R9 // R9: dst
SUBQ $4, R10
JS end
JZ one_crc
MOVQ R10, R13
SHRQ $2, R10 // len/4
ANDQ $3, R13 // len&3
XORQ BX, BX
ADDQ $1, R13
TESTQ R10, R10
JZ rem_loop
crc_loop:
MOVQ (R8), R11
XORQ BX, BX
XORQ DX, DX
XORQ DI, DI
MOVQ R11, R12
SHRQ $8, R11
MOVQ R12, AX
MOVQ R11, CX
SHRQ $16, R12
SHRQ $16, R11
MOVQ R12, SI
// CRC32 EAX, EBX
BYTE $0xF2; BYTE $0x0f
BYTE $0x38; BYTE $0xf1; BYTE $0xd8
// CRC32 ECX, EDX
BYTE $0xF2; BYTE $0x0f
BYTE $0x38; BYTE $0xf1; BYTE $0xd1
// CRC32 ESI, EDI
BYTE $0xF2; BYTE $0x0f
BYTE $0x38; BYTE $0xf1; BYTE $0xfe
MOVL BX, (R9)
MOVL DX, 4(R9)
MOVL DI, 8(R9)
XORQ BX, BX
MOVL R11, AX
// CRC32 EAX, EBX
BYTE $0xF2; BYTE $0x0f
BYTE $0x38; BYTE $0xf1; BYTE $0xd8
MOVL BX, 12(R9)
ADDQ $16, R9
ADDQ $4, R8
XORQ BX, BX
SUBQ $1, R10
JNZ crc_loop
rem_loop:
MOVL (R8), AX
// CRC32 EAX, EBX
BYTE $0xF2; BYTE $0x0f
BYTE $0x38; BYTE $0xf1; BYTE $0xd8
MOVL BX, (R9)
ADDQ $4, R9
ADDQ $1, R8
XORQ BX, BX
SUBQ $1, R13
JNZ rem_loop
end:
RET
one_crc:
MOVQ $1, R13
XORQ BX, BX
JMP rem_loop
// func matchLenSSE4(a, b []byte, max int) int
TEXT ·matchLenSSE4(SB), 4, $0
MOVQ a+0(FP), SI // RSI: &a
MOVQ b+24(FP), DI // RDI: &b
MOVQ max+48(FP), R10 // R10: max
XORQ R11, R11 // R11: match length
MOVQ R10, R12 // R12: Remainder
SHRQ $4, R10 // max / 16
MOVQ $16, AX // Set length for PCMPESTRI
MOVQ $16, DX // Set length for PCMPESTRI
ANDQ $15, R12 // max & 15
TESTQ R10, R10
JZ matchlen_verysmall
loopback_matchlen:
MOVOU (SI), X0 // a[x]
MOVOU (DI), X1 // b[x]
// PCMPESTRI $0x18, X1, X0
// 0x18 = _SIDD_UBYTE_OPS (0x0) | _SIDD_CMP_EQUAL_EACH (0x8) | _SIDD_NEGATIVE_POLARITY (0x10)
BYTE $0x66; BYTE $0x0f; BYTE $0x3a
BYTE $0x61; BYTE $0xc1; BYTE $0x18
JC match_ended
ADDQ $16, SI
ADDQ $16, DI
ADDQ $16, R11
SUBQ $1, R10
JNZ loopback_matchlen
// Check the remainder using REP CMPSB
matchlen_verysmall:
TESTQ R12, R12
JZ done_matchlen
MOVQ R12, CX
ADDQ R12, R11
// Compare CX bytes at [SI] [DI]
// Subtract one from CX for every match.
// Terminates when CX is zero (checked pre-compare)
CLD
REP; CMPSB
// Check if last was a match.
JZ done_matchlen
// Subtract remanding bytes.
SUBQ CX, R11
SUBQ $1, R11
MOVQ R11, ret+56(FP)
RET
match_ended:
ADDQ CX, R11
done_matchlen:
MOVQ R11, ret+56(FP)
RET
// func histogram(b []byte, h []int32)
TEXT ·histogram(SB), 4, $0
MOVQ b+0(FP), SI // SI: &b
MOVQ b_len+8(FP), R9 // R9: len(b)
MOVQ h+24(FP), DI // DI: Histogram
MOVQ R9, R8
SHRQ $3, R8
JZ hist1
XORQ R11, R11
loop_hist8:
MOVQ (SI), R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
MOVB R10, R11
INCL (DI)(R11*4)
SHRQ $8, R10
INCL (DI)(R10*4)
ADDQ $8, SI
DECQ R8
JNZ loop_hist8
hist1:
ANDQ $7, R9
JZ end_hist
XORQ R10, R10
loop_hist1:
MOVB (SI), R10
INCL (DI)(R10*4)
INCQ SI
DECQ R9
JNZ loop_hist1
end_hist:
RET

34
vendor/github.com/klauspost/compress/flate/crc32_noasm.go generated vendored Normal file
View File

@@ -0,0 +1,34 @@
//+build !amd64 noasm appengine
// Copyright 2015, Klaus Post, see LICENSE for details.
package flate
func init() {
useSSE42 = false
}
// crc32sse should never be called.
func crc32sse(a []byte) hash {
panic("no assembler")
}
// crc32sseAll should never be called.
func crc32sseAll(a []byte, dst []hash) {
panic("no assembler")
}
// matchLenSSE4 should never be called.
func matchLenSSE4(a, b []byte, max int) int {
panic("no assembler")
return 0
}
// histogram accumulates a histogram of b in h.
// h must be at least 256 entries in length,
// and must be cleared before calling this function.
func histogram(b []byte, h []int32) {
for _, t := range b {
h[t]++
}
}

1357
vendor/github.com/klauspost/compress/flate/deflate.go generated vendored Normal file
View File

File diff suppressed because it is too large Load Diff

632
vendor/github.com/klauspost/compress/flate/deflate_test.go generated vendored Normal file
View File

@@ -0,0 +1,632 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Copyright (c) 2015 Klaus Post
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
import (
"bytes"
"fmt"
"io"
"io/ioutil"
"reflect"
"strings"
"sync"
"testing"
)
type deflateTest struct {
in []byte
level int
out []byte
}
type deflateInflateTest struct {
in []byte
}
type reverseBitsTest struct {
in uint16
bitCount uint8
out uint16
}
var deflateTests = []*deflateTest{
{[]byte{}, 0, []byte{1, 0, 0, 255, 255}},
{[]byte{0x11}, BestCompression, []byte{18, 4, 4, 0, 0, 255, 255}},
{[]byte{0x11}, BestCompression, []byte{18, 4, 4, 0, 0, 255, 255}},
{[]byte{0x11}, BestCompression, []byte{18, 4, 4, 0, 0, 255, 255}},
{[]byte{0x11}, 0, []byte{0, 1, 0, 254, 255, 17, 1, 0, 0, 255, 255}},
{[]byte{0x11, 0x12}, 0, []byte{0, 2, 0, 253, 255, 17, 18, 1, 0, 0, 255, 255}},
{[]byte{0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11}, 0,
[]byte{0, 8, 0, 247, 255, 17, 17, 17, 17, 17, 17, 17, 17, 1, 0, 0, 255, 255},
},
{[]byte{}, 1, []byte{1, 0, 0, 255, 255}},
{[]byte{0x11}, BestCompression, []byte{18, 4, 4, 0, 0, 255, 255}},
{[]byte{0x11, 0x12}, BestCompression, []byte{18, 20, 2, 4, 0, 0, 255, 255}},
{[]byte{0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11}, BestCompression, []byte{18, 132, 2, 64, 0, 0, 0, 255, 255}},
{[]byte{}, 9, []byte{1, 0, 0, 255, 255}},
{[]byte{0x11}, 9, []byte{18, 4, 4, 0, 0, 255, 255}},
{[]byte{0x11, 0x12}, 9, []byte{18, 20, 2, 4, 0, 0, 255, 255}},
{[]byte{0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11}, 9, []byte{18, 132, 2, 64, 0, 0, 0, 255, 255}},
}
var deflateInflateTests = []*deflateInflateTest{
{[]byte{}},
{[]byte{0x11}},
{[]byte{0x11, 0x12}},
{[]byte{0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11, 0x11}},
{[]byte{0x11, 0x10, 0x13, 0x41, 0x21, 0x21, 0x41, 0x13, 0x87, 0x78, 0x13}},
{largeDataChunk()},
}
var reverseBitsTests = []*reverseBitsTest{
{1, 1, 1},
{1, 2, 2},
{1, 3, 4},
{1, 4, 8},
{1, 5, 16},
{17, 5, 17},
{257, 9, 257},
{29, 5, 23},
}
func largeDataChunk() []byte {
result := make([]byte, 100000)
for i := range result {
result[i] = byte(i * i & 0xFF)
}
return result
}
func TestCRCBulkOld(t *testing.T) {
for _, x := range deflateTests {
y := x.out
if len(y) >= minMatchLength {
y = append(y, y...)
for j := 4; j < len(y); j++ {
y := y[:j]
dst := make([]hash, len(y)-minMatchLength+1)
for i := range dst {
dst[i] = hash(i + 100)
}
oldBulkHash(y, dst)
for i, val := range dst {
got := val & hashMask
expect := oldHash(y[i:]) & hashMask
if got != expect && got == hash(i)+100 {
t.Errorf("Len:%d Index:%d, expected 0x%08x but not modified", len(y), i, expect)
} else if got != expect {
t.Errorf("Len:%d Index:%d, got 0x%08x expected:0x%08x", len(y), i, got, expect)
} else {
//t.Logf("Len:%d Index:%d OK (0x%08x)", len(y), i, got)
}
}
}
}
}
}
func TestDeflate(t *testing.T) {
for _, h := range deflateTests {
var buf bytes.Buffer
w, err := NewWriter(&buf, h.level)
if err != nil {
t.Errorf("NewWriter: %v", err)
continue
}
w.Write(h.in)
w.Close()
if !bytes.Equal(buf.Bytes(), h.out) {
t.Errorf("Deflate(%d, %x) = \n%#v, want \n%#v", h.level, h.in, buf.Bytes(), h.out)
}
}
}
// A sparseReader returns a stream consisting of 0s followed by 1<<16 1s.
// This tests missing hash references in a very large input.
type sparseReader struct {
l int64
cur int64
}
func (r *sparseReader) Read(b []byte) (n int, err error) {
if r.cur >= r.l {
return 0, io.EOF
}
n = len(b)
cur := r.cur + int64(n)
if cur > r.l {
n -= int(cur - r.l)
cur = r.l
}
for i := range b[0:n] {
if r.cur+int64(i) >= r.l-1<<16 {
b[i] = 1
} else {
b[i] = 0
}
}
r.cur = cur
return
}
func TestVeryLongSparseChunk(t *testing.T) {
if testing.Short() {
t.Skip("skipping sparse chunk during short test")
}
w, err := NewWriter(ioutil.Discard, 1)
if err != nil {
t.Errorf("NewWriter: %v", err)
return
}
if _, err = io.Copy(w, &sparseReader{l: 23E8}); err != nil {
t.Errorf("Compress failed: %v", err)
return
}
}
type syncBuffer struct {
buf bytes.Buffer
mu sync.RWMutex
closed bool
ready chan bool
}
func newSyncBuffer() *syncBuffer {
return &syncBuffer{ready: make(chan bool, 1)}
}
func (b *syncBuffer) Read(p []byte) (n int, err error) {
for {
b.mu.RLock()
n, err = b.buf.Read(p)
b.mu.RUnlock()
if n > 0 || b.closed {
return
}
<-b.ready
}
}
func (b *syncBuffer) signal() {
select {
case b.ready <- true:
default:
}
}
func (b *syncBuffer) Write(p []byte) (n int, err error) {
n, err = b.buf.Write(p)
b.signal()
return
}
func (b *syncBuffer) WriteMode() {
b.mu.Lock()
}
func (b *syncBuffer) ReadMode() {
b.mu.Unlock()
b.signal()
}
func (b *syncBuffer) Close() error {
b.closed = true
b.signal()
return nil
}
func testSync(t *testing.T, level int, input []byte, name string) {
if len(input) == 0 {
return
}
t.Logf("--testSync %d, %d, %s", level, len(input), name)
buf := newSyncBuffer()
buf1 := new(bytes.Buffer)
buf.WriteMode()
w, err := NewWriter(io.MultiWriter(buf, buf1), level)
if err != nil {
t.Errorf("NewWriter: %v", err)
return
}
r := NewReader(buf)
// Write half the input and read back.
for i := 0; i < 2; i++ {
var lo, hi int
if i == 0 {
lo, hi = 0, (len(input)+1)/2
} else {
lo, hi = (len(input)+1)/2, len(input)
}
t.Logf("#%d: write %d-%d", i, lo, hi)
if _, err := w.Write(input[lo:hi]); err != nil {
t.Errorf("testSync: write: %v", err)
return
}
if i == 0 {
if err := w.Flush(); err != nil {
t.Errorf("testSync: flush: %v", err)
return
}
} else {
if err := w.Close(); err != nil {
t.Errorf("testSync: close: %v", err)
}
}
buf.ReadMode()
out := make([]byte, hi-lo+1)
m, err := io.ReadAtLeast(r, out, hi-lo)
t.Logf("#%d: read %d", i, m)
if m != hi-lo || err != nil {
t.Errorf("testSync/%d (%d, %d, %s): read %d: %d, %v (%d left)", i, level, len(input), name, hi-lo, m, err, buf.buf.Len())
return
}
if !bytes.Equal(input[lo:hi], out[:hi-lo]) {
t.Errorf("testSync/%d: read wrong bytes: %x vs %x", i, input[lo:hi], out[:hi-lo])
return
}
// This test originally checked that after reading
// the first half of the input, there was nothing left
// in the read buffer (buf.buf.Len() != 0) but that is
// not necessarily the case: the write Flush may emit
// some extra framing bits that are not necessary
// to process to obtain the first half of the uncompressed
// data. The test ran correctly most of the time, because
// the background goroutine had usually read even
// those extra bits by now, but it's not a useful thing to
// check.
buf.WriteMode()
}
buf.ReadMode()
out := make([]byte, 10)
if n, err := r.Read(out); n > 0 || err != io.EOF {
t.Errorf("testSync (%d, %d, %s): final Read: %d, %v (hex: %x)", level, len(input), name, n, err, out[0:n])
}
if buf.buf.Len() != 0 {
t.Errorf("testSync (%d, %d, %s): extra data at end", level, len(input), name)
}
r.Close()
// stream should work for ordinary reader too
r = NewReader(buf1)
out, err = ioutil.ReadAll(r)
if err != nil {
t.Errorf("testSync: read: %s", err)
return
}
r.Close()
if !bytes.Equal(input, out) {
t.Errorf("testSync: decompress(compress(data)) != data: level=%d input=%s", level, name)
}
}
func testToFromWithLevelAndLimit(t *testing.T, level int, input []byte, name string, limit int) {
var buffer bytes.Buffer
w, err := NewWriter(&buffer, level)
if err != nil {
t.Errorf("NewWriter: %v", err)
return
}
w.Write(input)
w.Close()
if limit > 0 && buffer.Len() > limit {
t.Errorf("level: %d, len(compress(data)) = %d > limit = %d", level, buffer.Len(), limit)
return
}
if limit > 0 {
t.Logf("level: %d - Size:%.2f%%, %d b\n", level, float64(buffer.Len()*100)/float64(limit), buffer.Len())
}
r := NewReader(&buffer)
out, err := ioutil.ReadAll(r)
if err != nil {
t.Errorf("read: %s", err)
return
}
r.Close()
if !bytes.Equal(input, out) {
t.Errorf("decompress(compress(data)) != data: level=%d input=%s", level, name)
return
}
testSync(t, level, input, name)
}
func testToFromWithLimit(t *testing.T, input []byte, name string, limit [11]int) {
for i := 0; i < 10; i++ {
testToFromWithLevelAndLimit(t, i, input, name, limit[i])
}
testToFromWithLevelAndLimit(t, -2, input, name, limit[10])
}
func TestDeflateInflate(t *testing.T) {
for i, h := range deflateInflateTests {
testToFromWithLimit(t, h.in, fmt.Sprintf("#%d", i), [11]int{})
}
}
func TestReverseBits(t *testing.T) {
for _, h := range reverseBitsTests {
if v := reverseBits(h.in, h.bitCount); v != h.out {
t.Errorf("reverseBits(%v,%v) = %v, want %v",
h.in, h.bitCount, v, h.out)
}
}
}
type deflateInflateStringTest struct {
filename string
label string
limit [11]int // Number 11 is ConstantCompression
}
var deflateInflateStringTests = []deflateInflateStringTest{
{
"../testdata/e.txt",
"2.718281828...",
[...]int{100018, 67900, 50960, 51150, 50930, 50790, 50790, 50790, 50790, 50790, 43683 + 100},
},
{
"../testdata/Mark.Twain-Tom.Sawyer.txt",
"Mark.Twain-Tom.Sawyer",
[...]int{407330, 195000, 185361, 180974, 169160, 164476, 162936, 160506, 160295, 160295, 233460 + 100},
},
}
func TestDeflateInflateString(t *testing.T) {
for _, test := range deflateInflateStringTests {
gold, err := ioutil.ReadFile(test.filename)
if err != nil {
t.Error(err)
}
// Remove returns that may be present on Windows
neutral := strings.Map(func(r rune) rune {
if r != '\r' {
return r
}
return -1
}, string(gold))
testToFromWithLimit(t, []byte(neutral), test.label, test.limit)
if testing.Short() {
break
}
}
}
func TestReaderDict(t *testing.T) {
const (
dict = "hello world"
text = "hello again world"
)
var b bytes.Buffer
w, err := NewWriter(&b, 5)
if err != nil {
t.Fatalf("NewWriter: %v", err)
}
w.Write([]byte(dict))
w.Flush()
b.Reset()
w.Write([]byte(text))
w.Close()
r := NewReaderDict(&b, []byte(dict))
data, err := ioutil.ReadAll(r)
if err != nil {
t.Fatal(err)
}
if string(data) != "hello again world" {
t.Fatalf("read returned %q want %q", string(data), text)
}
}
func TestWriterDict(t *testing.T) {
const (
dict = "hello world Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua."
text = "hello world Lorem ipsum dolor sit amet"
)
// This test is sensitive to algorithm changes that skip
// data in favour of speed. Higher levels are less prone to this
// so we test level 4-9.
for l := 4; l < 9; l++ {
var b bytes.Buffer
w, err := NewWriter(&b, l)
if err != nil {
t.Fatalf("level %d, NewWriter: %v", l, err)
}
w.Write([]byte(dict))
w.Flush()
b.Reset()
w.Write([]byte(text))
w.Close()
var b1 bytes.Buffer
w, _ = NewWriterDict(&b1, l, []byte(dict))
w.Write([]byte(text))
w.Close()
if !bytes.Equal(b1.Bytes(), b.Bytes()) {
t.Errorf("level %d, writer wrote\n%v\n want\n%v", l, b1.Bytes(), b.Bytes())
}
}
}
// See http://code.google.com/p/go/issues/detail?id=2508
func TestRegression2508(t *testing.T) {
if testing.Short() {
t.Logf("test disabled with -short")
return
}
w, err := NewWriter(ioutil.Discard, 1)
if err != nil {
t.Fatalf("NewWriter: %v", err)
}
buf := make([]byte, 1024)
for i := 0; i < 131072; i++ {
if _, err := w.Write(buf); err != nil {
t.Fatalf("writer failed: %v", err)
}
}
w.Close()
}
func TestWriterReset(t *testing.T) {
for level := -2; level <= 9; level++ {
if level == -1 {
level++
}
if testing.Short() && level > 1 {
break
}
w, err := NewWriter(ioutil.Discard, level)
if err != nil {
t.Fatalf("NewWriter: %v", err)
}
buf := []byte("hello world")
for i := 0; i < 1024; i++ {
w.Write(buf)
}
w.Reset(ioutil.Discard)
wref, err := NewWriter(ioutil.Discard, level)
if err != nil {
t.Fatalf("NewWriter: %v", err)
}
// DeepEqual doesn't compare functions.
w.d.fill, wref.d.fill = nil, nil
w.d.step, wref.d.step = nil, nil
w.d.bulkHasher, wref.d.bulkHasher = nil, nil
w.d.snap, wref.d.snap = nil, nil
// hashMatch is always overwritten when used.
copy(w.d.hashMatch[:], wref.d.hashMatch[:])
if w.d.tokens.n != 0 {
t.Errorf("level %d Writer not reset after Reset. %d tokens were present", level, w.d.tokens.n)
}
// As long as the length is 0, we don't care about the content.
w.d.tokens = wref.d.tokens
// We don't care if there are values in the window, as long as it is at d.index is 0
w.d.window = wref.d.window
if !reflect.DeepEqual(w, wref) {
t.Errorf("level %d Writer not reset after Reset", level)
}
}
testResetOutput(t, func(w io.Writer) (*Writer, error) { return NewWriter(w, NoCompression) })
testResetOutput(t, func(w io.Writer) (*Writer, error) { return NewWriter(w, DefaultCompression) })
testResetOutput(t, func(w io.Writer) (*Writer, error) { return NewWriter(w, BestCompression) })
testResetOutput(t, func(w io.Writer) (*Writer, error) { return NewWriter(w, ConstantCompression) })
dict := []byte("we are the world")
testResetOutput(t, func(w io.Writer) (*Writer, error) { return NewWriterDict(w, NoCompression, dict) })
testResetOutput(t, func(w io.Writer) (*Writer, error) { return NewWriterDict(w, DefaultCompression, dict) })
testResetOutput(t, func(w io.Writer) (*Writer, error) { return NewWriterDict(w, BestCompression, dict) })
testResetOutput(t, func(w io.Writer) (*Writer, error) { return NewWriterDict(w, ConstantCompression, dict) })
}
func testResetOutput(t *testing.T, newWriter func(w io.Writer) (*Writer, error)) {
buf := new(bytes.Buffer)
w, err := newWriter(buf)
if err != nil {
t.Fatalf("NewWriter: %v", err)
}
b := []byte("hello world")
for i := 0; i < 1024; i++ {
w.Write(b)
}
w.Close()
out1 := buf.Bytes()
buf2 := new(bytes.Buffer)
w.Reset(buf2)
for i := 0; i < 1024; i++ {
w.Write(b)
}
w.Close()
out2 := buf2.Bytes()
if len(out1) != len(out2) {
t.Errorf("got %d, expected %d bytes", len(out2), len(out1))
}
if bytes.Compare(out1, out2) != 0 {
mm := 0
for i, b := range out1[:len(out2)] {
if b != out2[i] {
t.Errorf("mismatch index %d: %02x, expected %02x", i, out2[i], b)
}
mm++
if mm == 10 {
t.Fatal("Stopping")
}
}
}
t.Logf("got %d bytes", len(out1))
}
// A writer that fails after N writes.
type errorWriter struct {
N int
}
func (e *errorWriter) Write(b []byte) (int, error) {
if e.N <= 0 {
return 0, io.ErrClosedPipe
}
e.N--
return len(b), nil
}
// Test if errors from the underlying writer is passed upwards.
func TestWriteError(t *testing.T) {
buf := new(bytes.Buffer)
for i := 0; i < 1024*1024; i++ {
buf.WriteString(fmt.Sprintf("asdasfasf%d%dfghfgujyut%dyutyu\n", i, i, i))
}
in := buf.Bytes()
for l := -2; l < 10; l++ {
for fail := 1; fail <= 512; fail *= 2 {
// Fail after 2 writes
ew := &errorWriter{N: fail}
w, err := NewWriter(ew, l)
if err != nil {
t.Errorf("NewWriter: level %d: %v", l, err)
}
n, err := io.Copy(w, bytes.NewBuffer(in))
if err == nil {
t.Errorf("Level %d: Expected an error, writer was %#v", l, ew)
}
n2, err := w.Write([]byte{1, 2, 2, 3, 4, 5})
if n2 != 0 {
t.Error("Level", l, "Expected 0 length write, got", n)
}
if err == nil {
t.Error("Level", l, "Expected an error")
}
err = w.Flush()
if err == nil {
t.Error("Level", l, "Expected an error on close")
}
err = w.Close()
if err == nil {
t.Error("Level", l, "Expected an error on close")
}
w.Reset(ioutil.Discard)
n2, err = w.Write([]byte{1, 2, 3, 4, 5, 6})
if err != nil {
t.Error("Level", l, "Got unexpected error after reset:", err)
}
if n2 == 0 {
t.Error("Level", l, "Got 0 length write, expected > 0")
}
if testing.Short() {
return
}
}
}
}

78
vendor/github.com/klauspost/compress/flate/fixedhuff.go generated vendored Normal file
View File

@@ -0,0 +1,78 @@
// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
// autogenerated by go run gen.go -output fixedhuff.go, DO NOT EDIT
var fixedHuffmanDecoder = huffmanDecoder{
7,
[huffmanNumChunks]uint32{
0x1007, 0x0508, 0x0108, 0x1188, 0x1107, 0x0708, 0x0308, 0x0c09,
0x1087, 0x0608, 0x0208, 0x0a09, 0x0008, 0x0808, 0x0408, 0x0e09,
0x1047, 0x0588, 0x0188, 0x0909, 0x1147, 0x0788, 0x0388, 0x0d09,
0x10c7, 0x0688, 0x0288, 0x0b09, 0x0088, 0x0888, 0x0488, 0x0f09,
0x1027, 0x0548, 0x0148, 0x11c8, 0x1127, 0x0748, 0x0348, 0x0c89,
0x10a7, 0x0648, 0x0248, 0x0a89, 0x0048, 0x0848, 0x0448, 0x0e89,
0x1067, 0x05c8, 0x01c8, 0x0989, 0x1167, 0x07c8, 0x03c8, 0x0d89,
0x10e7, 0x06c8, 0x02c8, 0x0b89, 0x00c8, 0x08c8, 0x04c8, 0x0f89,
0x1017, 0x0528, 0x0128, 0x11a8, 0x1117, 0x0728, 0x0328, 0x0c49,
0x1097, 0x0628, 0x0228, 0x0a49, 0x0028, 0x0828, 0x0428, 0x0e49,
0x1057, 0x05a8, 0x01a8, 0x0949, 0x1157, 0x07a8, 0x03a8, 0x0d49,
0x10d7, 0x06a8, 0x02a8, 0x0b49, 0x00a8, 0x08a8, 0x04a8, 0x0f49,
0x1037, 0x0568, 0x0168, 0x11e8, 0x1137, 0x0768, 0x0368, 0x0cc9,
0x10b7, 0x0668, 0x0268, 0x0ac9, 0x0068, 0x0868, 0x0468, 0x0ec9,
0x1077, 0x05e8, 0x01e8, 0x09c9, 0x1177, 0x07e8, 0x03e8, 0x0dc9,
0x10f7, 0x06e8, 0x02e8, 0x0bc9, 0x00e8, 0x08e8, 0x04e8, 0x0fc9,
0x1007, 0x0518, 0x0118, 0x1198, 0x1107, 0x0718, 0x0318, 0x0c29,
0x1087, 0x0618, 0x0218, 0x0a29, 0x0018, 0x0818, 0x0418, 0x0e29,
0x1047, 0x0598, 0x0198, 0x0929, 0x1147, 0x0798, 0x0398, 0x0d29,
0x10c7, 0x0698, 0x0298, 0x0b29, 0x0098, 0x0898, 0x0498, 0x0f29,
0x1027, 0x0558, 0x0158, 0x11d8, 0x1127, 0x0758, 0x0358, 0x0ca9,
0x10a7, 0x0658, 0x0258, 0x0aa9, 0x0058, 0x0858, 0x0458, 0x0ea9,
0x1067, 0x05d8, 0x01d8, 0x09a9, 0x1167, 0x07d8, 0x03d8, 0x0da9,
0x10e7, 0x06d8, 0x02d8, 0x0ba9, 0x00d8, 0x08d8, 0x04d8, 0x0fa9,
0x1017, 0x0538, 0x0138, 0x11b8, 0x1117, 0x0738, 0x0338, 0x0c69,
0x1097, 0x0638, 0x0238, 0x0a69, 0x0038, 0x0838, 0x0438, 0x0e69,
0x1057, 0x05b8, 0x01b8, 0x0969, 0x1157, 0x07b8, 0x03b8, 0x0d69,
0x10d7, 0x06b8, 0x02b8, 0x0b69, 0x00b8, 0x08b8, 0x04b8, 0x0f69,
0x1037, 0x0578, 0x0178, 0x11f8, 0x1137, 0x0778, 0x0378, 0x0ce9,
0x10b7, 0x0678, 0x0278, 0x0ae9, 0x0078, 0x0878, 0x0478, 0x0ee9,
0x1077, 0x05f8, 0x01f8, 0x09e9, 0x1177, 0x07f8, 0x03f8, 0x0de9,
0x10f7, 0x06f8, 0x02f8, 0x0be9, 0x00f8, 0x08f8, 0x04f8, 0x0fe9,
0x1007, 0x0508, 0x0108, 0x1188, 0x1107, 0x0708, 0x0308, 0x0c19,
0x1087, 0x0608, 0x0208, 0x0a19, 0x0008, 0x0808, 0x0408, 0x0e19,
0x1047, 0x0588, 0x0188, 0x0919, 0x1147, 0x0788, 0x0388, 0x0d19,
0x10c7, 0x0688, 0x0288, 0x0b19, 0x0088, 0x0888, 0x0488, 0x0f19,
0x1027, 0x0548, 0x0148, 0x11c8, 0x1127, 0x0748, 0x0348, 0x0c99,
0x10a7, 0x0648, 0x0248, 0x0a99, 0x0048, 0x0848, 0x0448, 0x0e99,
0x1067, 0x05c8, 0x01c8, 0x0999, 0x1167, 0x07c8, 0x03c8, 0x0d99,
0x10e7, 0x06c8, 0x02c8, 0x0b99, 0x00c8, 0x08c8, 0x04c8, 0x0f99,
0x1017, 0x0528, 0x0128, 0x11a8, 0x1117, 0x0728, 0x0328, 0x0c59,
0x1097, 0x0628, 0x0228, 0x0a59, 0x0028, 0x0828, 0x0428, 0x0e59,
0x1057, 0x05a8, 0x01a8, 0x0959, 0x1157, 0x07a8, 0x03a8, 0x0d59,
0x10d7, 0x06a8, 0x02a8, 0x0b59, 0x00a8, 0x08a8, 0x04a8, 0x0f59,
0x1037, 0x0568, 0x0168, 0x11e8, 0x1137, 0x0768, 0x0368, 0x0cd9,
0x10b7, 0x0668, 0x0268, 0x0ad9, 0x0068, 0x0868, 0x0468, 0x0ed9,
0x1077, 0x05e8, 0x01e8, 0x09d9, 0x1177, 0x07e8, 0x03e8, 0x0dd9,
0x10f7, 0x06e8, 0x02e8, 0x0bd9, 0x00e8, 0x08e8, 0x04e8, 0x0fd9,
0x1007, 0x0518, 0x0118, 0x1198, 0x1107, 0x0718, 0x0318, 0x0c39,
0x1087, 0x0618, 0x0218, 0x0a39, 0x0018, 0x0818, 0x0418, 0x0e39,
0x1047, 0x0598, 0x0198, 0x0939, 0x1147, 0x0798, 0x0398, 0x0d39,
0x10c7, 0x0698, 0x0298, 0x0b39, 0x0098, 0x0898, 0x0498, 0x0f39,
0x1027, 0x0558, 0x0158, 0x11d8, 0x1127, 0x0758, 0x0358, 0x0cb9,
0x10a7, 0x0658, 0x0258, 0x0ab9, 0x0058, 0x0858, 0x0458, 0x0eb9,
0x1067, 0x05d8, 0x01d8, 0x09b9, 0x1167, 0x07d8, 0x03d8, 0x0db9,
0x10e7, 0x06d8, 0x02d8, 0x0bb9, 0x00d8, 0x08d8, 0x04d8, 0x0fb9,
0x1017, 0x0538, 0x0138, 0x11b8, 0x1117, 0x0738, 0x0338, 0x0c79,
0x1097, 0x0638, 0x0238, 0x0a79, 0x0038, 0x0838, 0x0438, 0x0e79,
0x1057, 0x05b8, 0x01b8, 0x0979, 0x1157, 0x07b8, 0x03b8, 0x0d79,
0x10d7, 0x06b8, 0x02b8, 0x0b79, 0x00b8, 0x08b8, 0x04b8, 0x0f79,
0x1037, 0x0578, 0x0178, 0x11f8, 0x1137, 0x0778, 0x0378, 0x0cf9,
0x10b7, 0x0678, 0x0278, 0x0af9, 0x0078, 0x0878, 0x0478, 0x0ef9,
0x1077, 0x05f8, 0x01f8, 0x09f9, 0x1177, 0x07f8, 0x03f8, 0x0df9,
0x10f7, 0x06f8, 0x02f8, 0x0bf9, 0x00f8, 0x08f8, 0x04f8, 0x0ff9,
},
nil, 0,
}

260
vendor/github.com/klauspost/compress/flate/flate_test.go generated vendored Normal file
View File

@@ -0,0 +1,260 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// This test tests some internals of the flate package.
// The tests in package compress/gzip serve as the
// end-to-end test of the decompressor.
package flate
import (
"bytes"
"encoding/hex"
"io/ioutil"
"testing"
)
// The following test should not panic.
func TestIssue5915(t *testing.T) {
bits := []int{4, 0, 0, 6, 4, 3, 2, 3, 3, 4, 4, 5, 0, 0, 0, 0, 5, 5, 6,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 11, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 8, 6, 0, 11, 0, 8, 0, 6, 6, 10, 8}
var h huffmanDecoder
if h.init(bits) {
t.Fatalf("Given sequence of bits is bad, and should not succeed.")
}
}
// The following test should not panic.
func TestIssue5962(t *testing.T) {
bits := []int{4, 0, 0, 6, 4, 3, 2, 3, 3, 4, 4, 5, 0, 0, 0, 0,
5, 5, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 11}
var h huffmanDecoder
if h.init(bits) {
t.Fatalf("Given sequence of bits is bad, and should not succeed.")
}
}
// The following test should not panic.
func TestIssue6255(t *testing.T) {
bits1 := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 11}
bits2 := []int{11, 13}
var h huffmanDecoder
if !h.init(bits1) {
t.Fatalf("Given sequence of bits is good and should succeed.")
}
if h.init(bits2) {
t.Fatalf("Given sequence of bits is bad and should not succeed.")
}
}
func TestInvalidEncoding(t *testing.T) {
// Initialize Huffman decoder to recognize "0".
var h huffmanDecoder
if !h.init([]int{1}) {
t.Fatal("Failed to initialize Huffman decoder")
}
// Initialize decompressor with invalid Huffman coding.
var f decompressor
f.r = bytes.NewReader([]byte{0xff})
_, err := f.huffSym(&h)
if err == nil {
t.Fatal("Should have rejected invalid bit sequence")
}
}
func TestInvalidBits(t *testing.T) {
oversubscribed := []int{1, 2, 3, 4, 4, 5}
incomplete := []int{1, 2, 4, 4}
var h huffmanDecoder
if h.init(oversubscribed) {
t.Fatal("Should reject oversubscribed bit-length set")
}
if h.init(incomplete) {
t.Fatal("Should reject incomplete bit-length set")
}
}
func TestStreams(t *testing.T) {
// To verify any of these hexstrings as valid or invalid flate streams
// according to the C zlib library, you can use the Python wrapper library:
// >>> hex_string = "010100feff11"
// >>> import zlib
// >>> zlib.decompress(hex_string.decode("hex"), -15) # Negative means raw DEFLATE
// '\x11'
testCases := []struct {
desc string // Description of the stream
stream string // Hexstring of the input DEFLATE stream
want string // Expected result. Use "fail" to expect failure
}{{
"degenerate HCLenTree",
"05e0010000000000100000000000000000000000000000000000000000000000" +
"00000000000000000004",
"fail",
}, {
"complete HCLenTree, empty HLitTree, empty HDistTree",
"05e0010400000000000000000000000000000000000000000000000000000000" +
"00000000000000000010",
"fail",
}, {
"empty HCLenTree",
"05e0010000000000000000000000000000000000000000000000000000000000" +
"00000000000000000010",
"fail",
}, {
"complete HCLenTree, complete HLitTree, empty HDistTree, use missing HDist symbol",
"000100feff000de0010400000000100000000000000000000000000000000000" +
"0000000000000000000000000000002c",
"fail",
}, {
"complete HCLenTree, complete HLitTree, degenerate HDistTree, use missing HDist symbol",
"000100feff000de0010000000000000000000000000000000000000000000000" +
"00000000000000000610000000004070",
"fail",
}, {
"complete HCLenTree, empty HLitTree, empty HDistTree",
"05e0010400000000100400000000000000000000000000000000000000000000" +
"0000000000000000000000000008",
"fail",
}, {
"complete HCLenTree, empty HLitTree, degenerate HDistTree",
"05e0010400000000100400000000000000000000000000000000000000000000" +
"0000000000000000000800000008",
"fail",
}, {
"complete HCLenTree, degenerate HLitTree, degenerate HDistTree, use missing HLit symbol",
"05e0010400000000100000000000000000000000000000000000000000000000" +
"0000000000000000001c",
"fail",
}, {
"complete HCLenTree, complete HLitTree, too large HDistTree",
"edff870500000000200400000000000000000000000000000000000000000000" +
"000000000000000000080000000000000004",
"fail",
}, {
"complete HCLenTree, complete HLitTree, empty HDistTree, excessive repeater code",
"edfd870500000000200400000000000000000000000000000000000000000000" +
"000000000000000000e8b100",
"fail",
}, {
"complete HCLenTree, complete HLitTree, empty HDistTree of normal length 30",
"05fd01240000000000f8ffffffffffffffffffffffffffffffffffffffffffff" +
"ffffffffffffffffff07000000fe01",
"",
}, {
"complete HCLenTree, complete HLitTree, empty HDistTree of excessive length 31",
"05fe01240000000000f8ffffffffffffffffffffffffffffffffffffffffffff" +
"ffffffffffffffffff07000000fc03",
"fail",
}, {
"complete HCLenTree, over-subscribed HLitTree, empty HDistTree",
"05e001240000000000fcffffffffffffffffffffffffffffffffffffffffffff" +
"ffffffffffffffffff07f00f",
"fail",
}, {
"complete HCLenTree, under-subscribed HLitTree, empty HDistTree",
"05e001240000000000fcffffffffffffffffffffffffffffffffffffffffffff" +
"fffffffffcffffffff07f00f",
"fail",
}, {
"complete HCLenTree, complete HLitTree with single code, empty HDistTree",
"05e001240000000000f8ffffffffffffffffffffffffffffffffffffffffffff" +
"ffffffffffffffffff07f00f",
"01",
}, {
"complete HCLenTree, complete HLitTree with multiple codes, empty HDistTree",
"05e301240000000000f8ffffffffffffffffffffffffffffffffffffffffffff" +
"ffffffffffffffffff07807f",
"01",
}, {
"complete HCLenTree, complete HLitTree, degenerate HDistTree, use valid HDist symbol",
"000100feff000de0010400000000100000000000000000000000000000000000" +
"0000000000000000000000000000003c",
"00000000",
}, {
"complete HCLenTree, degenerate HLitTree, degenerate HDistTree",
"05e0010400000000100000000000000000000000000000000000000000000000" +
"0000000000000000000c",
"",
}, {
"complete HCLenTree, degenerate HLitTree, empty HDistTree",
"05e0010400000000100000000000000000000000000000000000000000000000" +
"00000000000000000004",
"",
}, {
"complete HCLenTree, complete HLitTree, empty HDistTree, spanning repeater code",
"edfd870500000000200400000000000000000000000000000000000000000000" +
"000000000000000000e8b000",
"",
}, {
"complete HCLenTree with length codes, complete HLitTree, empty HDistTree",
"ede0010400000000100000000000000000000000000000000000000000000000" +
"0000000000000000000400004000",
"",
}, {
"complete HCLenTree, complete HLitTree, degenerate HDistTree, use valid HLit symbol 284 with count 31",
"000100feff00ede0010400000000100000000000000000000000000000000000" +
"000000000000000000000000000000040000407f00",
"0000000000000000000000000000000000000000000000000000000000000000" +
"0000000000000000000000000000000000000000000000000000000000000000" +
"0000000000000000000000000000000000000000000000000000000000000000" +
"0000000000000000000000000000000000000000000000000000000000000000" +
"0000000000000000000000000000000000000000000000000000000000000000" +
"0000000000000000000000000000000000000000000000000000000000000000" +
"0000000000000000000000000000000000000000000000000000000000000000" +
"0000000000000000000000000000000000000000000000000000000000000000" +
"000000",
}, {
"complete HCLenTree, complete HLitTree, degenerate HDistTree, use valid HLit and HDist symbols",
"0cc2010d00000082b0ac4aff0eb07d27060000ffff",
"616263616263",
}, {
"fixed block, use reserved symbol 287",
"33180700",
"fail",
}, {
"raw block",
"010100feff11",
"11",
}, {
"issue 10426 - over-subscribed HCLenTree causes a hang",
"344c4a4e494d4b070000ff2e2eff2e2e2e2e2eff",
"fail",
}, {
"issue 11030 - empty HDistTree unexpectedly leads to error",
"05c0070600000080400fff37a0ca",
"",
}, {
"issue 11033 - empty HDistTree unexpectedly leads to error",
"050fb109c020cca5d017dcbca044881ee1034ec149c8980bbc413c2ab35be9dc" +
"b1473449922449922411202306ee97b0383a521b4ffdcf3217f9f7d3adb701",
"3130303634342068652e706870005d05355f7ed957ff084a90925d19e3ebc6d0" +
"c6d7",
}}
for i, tc := range testCases {
data, err := hex.DecodeString(tc.stream)
if err != nil {
t.Fatal(err)
}
data, err = ioutil.ReadAll(NewReader(bytes.NewReader(data)))
if tc.want == "fail" {
if err == nil {
t.Errorf("#%d (%s): got nil error, want non-nil", i, tc.desc)
}
} else {
if err != nil {
t.Errorf("#%d (%s): %v", i, tc.desc, err)
continue
}
if got := hex.EncodeToString(data); got != tc.want {
t.Errorf("#%d (%s):\ngot %q\nwant %q", i, tc.desc, got, tc.want)
}
}
}
}

265
vendor/github.com/klauspost/compress/flate/gen.go generated vendored Normal file
View File

@@ -0,0 +1,265 @@
// Copyright 2012 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// +build ignore
// This program generates fixedhuff.go
// Invoke as
//
// go run gen.go -output fixedhuff.go
package main
import (
"bytes"
"flag"
"fmt"
"go/format"
"io/ioutil"
"log"
)
var filename = flag.String("output", "fixedhuff.go", "output file name")
const maxCodeLen = 16
// Note: the definition of the huffmanDecoder struct is copied from
// inflate.go, as it is private to the implementation.
// chunk & 15 is number of bits
// chunk >> 4 is value, including table link
const (
huffmanChunkBits = 9
huffmanNumChunks = 1 << huffmanChunkBits
huffmanCountMask = 15
huffmanValueShift = 4
)
type huffmanDecoder struct {
min int // the minimum code length
chunks [huffmanNumChunks]uint32 // chunks as described above
links [][]uint32 // overflow links
linkMask uint32 // mask the width of the link table
}
// Initialize Huffman decoding tables from array of code lengths.
// Following this function, h is guaranteed to be initialized into a complete
// tree (i.e., neither over-subscribed nor under-subscribed). The exception is a
// degenerate case where the tree has only a single symbol with length 1. Empty
// trees are permitted.
func (h *huffmanDecoder) init(bits []int) bool {
// Sanity enables additional runtime tests during Huffman
// table construction. It's intended to be used during
// development to supplement the currently ad-hoc unit tests.
const sanity = false
if h.min != 0 {
*h = huffmanDecoder{}
}
// Count number of codes of each length,
// compute min and max length.
var count [maxCodeLen]int
var min, max int
for _, n := range bits {
if n == 0 {
continue
}
if min == 0 || n < min {
min = n
}
if n > max {
max = n
}
count[n]++
}
// Empty tree. The decompressor.huffSym function will fail later if the tree
// is used. Technically, an empty tree is only valid for the HDIST tree and
// not the HCLEN and HLIT tree. However, a stream with an empty HCLEN tree
// is guaranteed to fail since it will attempt to use the tree to decode the
// codes for the HLIT and HDIST trees. Similarly, an empty HLIT tree is
// guaranteed to fail later since the compressed data section must be
// composed of at least one symbol (the end-of-block marker).
if max == 0 {
return true
}
code := 0
var nextcode [maxCodeLen]int
for i := min; i <= max; i++ {
code <<= 1
nextcode[i] = code
code += count[i]
}
// Check that the coding is complete (i.e., that we've
// assigned all 2-to-the-max possible bit sequences).
// Exception: To be compatible with zlib, we also need to
// accept degenerate single-code codings. See also
// TestDegenerateHuffmanCoding.
if code != 1<<uint(max) && !(code == 1 && max == 1) {
return false
}
h.min = min
if max > huffmanChunkBits {
numLinks := 1 << (uint(max) - huffmanChunkBits)
h.linkMask = uint32(numLinks - 1)
// create link tables
link := nextcode[huffmanChunkBits+1] >> 1
h.links = make([][]uint32, huffmanNumChunks-link)
for j := uint(link); j < huffmanNumChunks; j++ {
reverse := int(reverseByte[j>>8]) | int(reverseByte[j&0xff])<<8
reverse >>= uint(16 - huffmanChunkBits)
off := j - uint(link)
if sanity && h.chunks[reverse] != 0 {
panic("impossible: overwriting existing chunk")
}
h.chunks[reverse] = uint32(off<<huffmanValueShift | (huffmanChunkBits + 1))
h.links[off] = make([]uint32, numLinks)
}
}
for i, n := range bits {
if n == 0 {
continue
}
code := nextcode[n]
nextcode[n]++
chunk := uint32(i<<huffmanValueShift | n)
reverse := int(reverseByte[code>>8]) | int(reverseByte[code&0xff])<<8
reverse >>= uint(16 - n)
if n <= huffmanChunkBits {
for off := reverse; off < len(h.chunks); off += 1 << uint(n) {
// We should never need to overwrite
// an existing chunk. Also, 0 is
// never a valid chunk, because the
// lower 4 "count" bits should be
// between 1 and 15.
if sanity && h.chunks[off] != 0 {
panic("impossible: overwriting existing chunk")
}
h.chunks[off] = chunk
}
} else {
j := reverse & (huffmanNumChunks - 1)
if sanity && h.chunks[j]&huffmanCountMask != huffmanChunkBits+1 {
// Longer codes should have been
// associated with a link table above.
panic("impossible: not an indirect chunk")
}
value := h.chunks[j] >> huffmanValueShift
linktab := h.links[value]
reverse >>= huffmanChunkBits
for off := reverse; off < len(linktab); off += 1 << uint(n-huffmanChunkBits) {
if sanity && linktab[off] != 0 {
panic("impossible: overwriting existing chunk")
}
linktab[off] = chunk
}
}
}
if sanity {
// Above we've sanity checked that we never overwrote
// an existing entry. Here we additionally check that
// we filled the tables completely.
for i, chunk := range h.chunks {
if chunk == 0 {
// As an exception, in the degenerate
// single-code case, we allow odd
// chunks to be missing.
if code == 1 && i%2 == 1 {
continue
}
panic("impossible: missing chunk")
}
}
for _, linktab := range h.links {
for _, chunk := range linktab {
if chunk == 0 {
panic("impossible: missing chunk")
}
}
}
}
return true
}
func main() {
flag.Parse()
var h huffmanDecoder
var bits [288]int
initReverseByte()
for i := 0; i < 144; i++ {
bits[i] = 8
}
for i := 144; i < 256; i++ {
bits[i] = 9
}
for i := 256; i < 280; i++ {
bits[i] = 7
}
for i := 280; i < 288; i++ {
bits[i] = 8
}
h.init(bits[:])
if h.links != nil {
log.Fatal("Unexpected links table in fixed Huffman decoder")
}
var buf bytes.Buffer
fmt.Fprintf(&buf, `// Copyright 2013 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.`+"\n\n")
fmt.Fprintln(&buf, "package flate")
fmt.Fprintln(&buf)
fmt.Fprintln(&buf, "// autogenerated by go run gen.go -output fixedhuff.go, DO NOT EDIT")
fmt.Fprintln(&buf)
fmt.Fprintln(&buf, "var fixedHuffmanDecoder = huffmanDecoder{")
fmt.Fprintf(&buf, "\t%d,\n", h.min)
fmt.Fprintln(&buf, "\t[huffmanNumChunks]uint32{")
for i := 0; i < huffmanNumChunks; i++ {
if i&7 == 0 {
fmt.Fprintf(&buf, "\t\t")
} else {
fmt.Fprintf(&buf, " ")
}
fmt.Fprintf(&buf, "0x%04x,", h.chunks[i])
if i&7 == 7 {
fmt.Fprintln(&buf)
}
}
fmt.Fprintln(&buf, "\t},")
fmt.Fprintln(&buf, "\tnil, 0,")
fmt.Fprintln(&buf, "}")
data, err := format.Source(buf.Bytes())
if err != nil {
log.Fatal(err)
}
err = ioutil.WriteFile(*filename, data, 0644)
if err != nil {
log.Fatal(err)
}
}
var reverseByte [256]byte
func initReverseByte() {
for x := 0; x < 256; x++ {
var result byte
for i := uint(0); i < 8; i++ {
result |= byte(((x >> i) & 1) << (7 - i))
}
reverseByte[x] = result
}
}

717
vendor/github.com/klauspost/compress/flate/huffman_bit_writer.go generated vendored Normal file
View File

@@ -0,0 +1,717 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
import (
"io"
"math"
)
const (
// The largest offset code.
offsetCodeCount = 30
// The special code used to mark the end of a block.
endBlockMarker = 256
// The first length code.
lengthCodesStart = 257
// The number of codegen codes.
codegenCodeCount = 19
badCode = 255
// Output byte buffer size
// Must be multiple of 6 (48 bits) + 8
bufferSize = 240 + 8
)
// The number of extra bits needed by length code X - LENGTH_CODES_START.
var lengthExtraBits = []int8{
/* 257 */ 0, 0, 0,
/* 260 */ 0, 0, 0, 0, 0, 1, 1, 1, 1, 2,
/* 270 */ 2, 2, 2, 3, 3, 3, 3, 4, 4, 4,
/* 280 */ 4, 5, 5, 5, 5, 0,
}
// The length indicated by length code X - LENGTH_CODES_START.
var lengthBase = []uint32{
0, 1, 2, 3, 4, 5, 6, 7, 8, 10,
12, 14, 16, 20, 24, 28, 32, 40, 48, 56,
64, 80, 96, 112, 128, 160, 192, 224, 255,
}
// offset code word extra bits.
var offsetExtraBits = []int8{
0, 0, 0, 0, 1, 1, 2, 2, 3, 3,
4, 4, 5, 5, 6, 6, 7, 7, 8, 8,
9, 9, 10, 10, 11, 11, 12, 12, 13, 13,
/* extended window */
14, 14, 15, 15, 16, 16, 17, 17, 18, 18, 19, 19, 20, 20,
}
var offsetBase = []uint32{
/* normal deflate */
0x000000, 0x000001, 0x000002, 0x000003, 0x000004,
0x000006, 0x000008, 0x00000c, 0x000010, 0x000018,
0x000020, 0x000030, 0x000040, 0x000060, 0x000080,
0x0000c0, 0x000100, 0x000180, 0x000200, 0x000300,
0x000400, 0x000600, 0x000800, 0x000c00, 0x001000,
0x001800, 0x002000, 0x003000, 0x004000, 0x006000,
/* extended window */
0x008000, 0x00c000, 0x010000, 0x018000, 0x020000,
0x030000, 0x040000, 0x060000, 0x080000, 0x0c0000,
0x100000, 0x180000, 0x200000, 0x300000,
}
// The odd order in which the codegen code sizes are written.
var codegenOrder = []uint32{16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15}
type huffmanBitWriter struct {
w io.Writer
// Data waiting to be written is bytes[0:nbytes]
// and then the low nbits of bits.
bits uint64
nbits uint
bytes [bufferSize]byte
nbytes int
literalFreq []int32
offsetFreq []int32
codegen []uint8
codegenFreq []int32
literalEncoding *huffmanEncoder
offsetEncoding *huffmanEncoder
codegenEncoding *huffmanEncoder
err error
}
func newHuffmanBitWriter(w io.Writer) *huffmanBitWriter {
return &huffmanBitWriter{
w: w,
literalFreq: make([]int32, maxNumLit),
offsetFreq: make([]int32, offsetCodeCount),
codegen: make([]uint8, maxNumLit+offsetCodeCount+1),
codegenFreq: make([]int32, codegenCodeCount),
literalEncoding: newHuffmanEncoder(maxNumLit),
codegenEncoding: newHuffmanEncoder(codegenCodeCount),
offsetEncoding: newHuffmanEncoder(offsetCodeCount),
}
}
func (w *huffmanBitWriter) reset(writer io.Writer) {
w.w = writer
w.bits, w.nbits, w.nbytes, w.err = 0, 0, 0, nil
w.bytes = [bufferSize]byte{}
}
func (w *huffmanBitWriter) flush() {
if w.err != nil {
w.nbits = 0
return
}
n := w.nbytes
for w.nbits != 0 {
w.bytes[n] = byte(w.bits)
w.bits >>= 8
if w.nbits > 8 { // Avoid underflow
w.nbits -= 8
} else {
w.nbits = 0
}
n++
}
w.bits = 0
_, w.err = w.w.Write(w.bytes[0:n])
w.nbytes = 0
}
func (w *huffmanBitWriter) writeBits(b int32, nb uint) {
w.bits |= uint64(b) << w.nbits
w.nbits += nb
if w.nbits >= 48 {
bits := w.bits
w.bits >>= 48
w.nbits -= 48
n := w.nbytes
w.bytes[n] = byte(bits)
w.bytes[n+1] = byte(bits >> 8)
w.bytes[n+2] = byte(bits >> 16)
w.bytes[n+3] = byte(bits >> 24)
w.bytes[n+4] = byte(bits >> 32)
w.bytes[n+5] = byte(bits >> 40)
n += 6
if n >= bufferSize-8 {
_, w.err = w.w.Write(w.bytes[:bufferSize-8])
n = 0
}
w.nbytes = n
}
}
func (w *huffmanBitWriter) writeBytes(bytes []byte) {
if w.err != nil {
return
}
n := w.nbytes
for w.nbits != 0 {
w.bytes[n] = byte(w.bits)
w.bits >>= 8
w.nbits -= 8
n++
}
if w.nbits != 0 {
w.err = InternalError("writeBytes with unfinished bits")
return
}
if n != 0 {
_, w.err = w.w.Write(w.bytes[0:n])
if w.err != nil {
return
}
}
w.nbytes = 0
_, w.err = w.w.Write(bytes)
}
// RFC 1951 3.2.7 specifies a special run-length encoding for specifying
// the literal and offset lengths arrays (which are concatenated into a single
// array). This method generates that run-length encoding.
//
// The result is written into the codegen array, and the frequencies
// of each code is written into the codegenFreq array.
// Codes 0-15 are single byte codes. Codes 16-18 are followed by additional
// information. Code badCode is an end marker
//
// numLiterals The number of literals in literalEncoding
// numOffsets The number of offsets in offsetEncoding
func (w *huffmanBitWriter) generateCodegen(numLiterals int, numOffsets int, offenc *huffmanEncoder) {
for i := range w.codegenFreq {
w.codegenFreq[i] = 0
}
// Note that we are using codegen both as a temporary variable for holding
// a copy of the frequencies, and as the place where we put the result.
// This is fine because the output is always shorter than the input used
// so far.
codegen := w.codegen // cache
// Copy the concatenated code sizes to codegen. Put a marker at the end.
cgnl := codegen[0:numLiterals]
for i := range cgnl {
cgnl[i] = uint8(w.literalEncoding.codes[i].bits())
}
cgnl = codegen[numLiterals : numLiterals+numOffsets]
for i := range cgnl {
cgnl[i] = uint8(offenc.codes[i].bits())
}
codegen[numLiterals+numOffsets] = badCode
size := codegen[0]
count := 1
outIndex := 0
for inIndex := 1; size != badCode; inIndex++ {
// INVARIANT: We have seen "count" copies of size that have not yet
// had output generated for them.
nextSize := codegen[inIndex]
if nextSize == size {
count++
continue
}
// We need to generate codegen indicating "count" of size.
if size != 0 {
codegen[outIndex] = size
outIndex++
w.codegenFreq[size]++
count--
for count >= 3 {
n := 6
if n > count {
n = count
}
codegen[outIndex] = 16
outIndex++
codegen[outIndex] = uint8(n - 3)
outIndex++
w.codegenFreq[16]++
count -= n
}
} else {
for count >= 11 {
n := 138
if n > count {
n = count
}
codegen[outIndex] = 18
outIndex++
codegen[outIndex] = uint8(n - 11)
outIndex++
w.codegenFreq[18]++
count -= n
}
if count >= 3 {
// count >= 3 && count <= 10
codegen[outIndex] = 17
outIndex++
codegen[outIndex] = uint8(count - 3)
outIndex++
w.codegenFreq[17]++
count = 0
}
}
count--
for ; count >= 0; count-- {
codegen[outIndex] = size
outIndex++
w.codegenFreq[size]++
}
// Set up invariant for next time through the loop.
size = nextSize
count = 1
}
// Marker indicating the end of the codegen.
codegen[outIndex] = badCode
}
func (w *huffmanBitWriter) writeCode(c hcode) {
if w.err != nil {
return
}
w.bits |= uint64(c.code()) << w.nbits
w.nbits += c.bits()
if w.nbits >= 48 {
bits := w.bits
w.bits >>= 48
w.nbits -= 48
n := w.nbytes
w.bytes[n] = byte(bits)
w.bytes[n+1] = byte(bits >> 8)
w.bytes[n+2] = byte(bits >> 16)
w.bytes[n+3] = byte(bits >> 24)
w.bytes[n+4] = byte(bits >> 32)
w.bytes[n+5] = byte(bits >> 40)
n += 6
if n >= bufferSize-8 {
_, w.err = w.w.Write(w.bytes[:bufferSize-8])
n = 0
}
w.nbytes = n
}
}
// Write the header of a dynamic Huffman block to the output stream.
//
// numLiterals The number of literals specified in codegen
// numOffsets The number of offsets specified in codegen
// numCodegens The number of codegens used in codegen
func (w *huffmanBitWriter) writeDynamicHeader(numLiterals int, numOffsets int, numCodegens int, isEof bool) {
if w.err != nil {
return
}
var firstBits int32 = 4
if isEof {
firstBits = 5
}
w.writeBits(firstBits, 3)
w.writeBits(int32(numLiterals-257), 5)
w.writeBits(int32(numOffsets-1), 5)
w.writeBits(int32(numCodegens-4), 4)
for i := 0; i < numCodegens; i++ {
value := w.codegenEncoding.codes[codegenOrder[i]].bits()
w.writeBits(int32(value), 3)
}
i := 0
for {
var codeWord int = int(w.codegen[i])
i++
if codeWord == badCode {
break
}
// The low byte contains the actual code to generate.
w.writeCode(w.codegenEncoding.codes[uint32(codeWord)])
switch codeWord {
case 16:
w.writeBits(int32(w.codegen[i]), 2)
i++
break
case 17:
w.writeBits(int32(w.codegen[i]), 3)
i++
break
case 18:
w.writeBits(int32(w.codegen[i]), 7)
i++
break
}
}
}
func (w *huffmanBitWriter) writeStoredHeader(length int, isEof bool) {
if w.err != nil {
return
}
var flag int32
if isEof {
flag = 1
}
w.writeBits(flag, 3)
w.flush()
w.writeBits(int32(length), 16)
w.writeBits(int32(^uint16(length)), 16)
}
func (w *huffmanBitWriter) writeFixedHeader(isEof bool) {
if w.err != nil {
return
}
// Indicate that we are a fixed Huffman block
var value int32 = 2
if isEof {
value = 3
}
w.writeBits(value, 3)
}
func (w *huffmanBitWriter) writeBlock(tok tokens, eof bool, input []byte) {
if w.err != nil {
return
}
for i := range w.literalFreq {
w.literalFreq[i] = 0
}
for i := range w.offsetFreq {
w.offsetFreq[i] = 0
}
tok.tokens[tok.n] = endBlockMarker
tokens := tok.tokens[0 : tok.n+1]
for _, t := range tokens {
if t < matchType {
w.literalFreq[t.literal()]++
} else {
length := t.length()
offset := t.offset()
w.literalFreq[lengthCodesStart+lengthCode(length)]++
w.offsetFreq[offsetCode(offset)]++
}
}
// get the number of literals
numLiterals := len(w.literalFreq)
for w.literalFreq[numLiterals-1] == 0 {
numLiterals--
}
// get the number of offsets
numOffsets := len(w.offsetFreq)
for numOffsets > 0 && w.offsetFreq[numOffsets-1] == 0 {
numOffsets--
}
if numOffsets == 0 {
// We haven't found a single match. If we want to go with the dynamic encoding,
// we should count at least one offset to be sure that the offset huffman tree could be encoded.
w.offsetFreq[0] = 1
numOffsets = 1
}
w.literalEncoding.generate(w.literalFreq, 15)
w.offsetEncoding.generate(w.offsetFreq, 15)
storedBytes := 0
if input != nil {
storedBytes = len(input)
}
var extraBits int64
var storedSize int64 = math.MaxInt64
if storedBytes <= maxStoreBlockSize && input != nil {
storedSize = int64((storedBytes + 5) * 8)
// We only bother calculating the costs of the extra bits required by
// the length of offset fields (which will be the same for both fixed
// and dynamic encoding), if we need to compare those two encodings
// against stored encoding.
for lengthCode := lengthCodesStart + 8; lengthCode < numLiterals; lengthCode++ {
// First eight length codes have extra size = 0.
extraBits += int64(w.literalFreq[lengthCode]) * int64(lengthExtraBits[lengthCode-lengthCodesStart])
}
for offsetCode := 4; offsetCode < numOffsets; offsetCode++ {
// First four offset codes have extra size = 0.
extraBits += int64(w.offsetFreq[offsetCode]) * int64(offsetExtraBits[offsetCode])
}
}
// Figure out smallest code.
// Fixed Huffman baseline.
var size = int64(3) +
fixedLiteralEncoding.bitLength(w.literalFreq) +
fixedOffsetEncoding.bitLength(w.offsetFreq) +
extraBits
var literalEncoding = fixedLiteralEncoding
var offsetEncoding = fixedOffsetEncoding
// Dynamic Huffman?
var numCodegens int
// Generate codegen and codegenFrequencies, which indicates how to encode
// the literalEncoding and the offsetEncoding.
w.generateCodegen(numLiterals, numOffsets, w.offsetEncoding)
w.codegenEncoding.generate(w.codegenFreq, 7)
numCodegens = len(w.codegenFreq)
for numCodegens > 4 && w.codegenFreq[codegenOrder[numCodegens-1]] == 0 {
numCodegens--
}
dynamicHeader := int64(3+5+5+4+(3*numCodegens)) +
w.codegenEncoding.bitLength(w.codegenFreq) +
int64(extraBits) +
int64(w.codegenFreq[16]*2) +
int64(w.codegenFreq[17]*3) +
int64(w.codegenFreq[18]*7)
dynamicSize := dynamicHeader +
w.literalEncoding.bitLength(w.literalFreq) +
w.offsetEncoding.bitLength(w.offsetFreq)
if dynamicSize < size {
size = dynamicSize
literalEncoding = w.literalEncoding
offsetEncoding = w.offsetEncoding
}
// Stored bytes?
if storedSize < size {
w.writeStoredHeader(storedBytes, eof)
w.writeBytes(input[0:storedBytes])
return
}
// Huffman.
if literalEncoding == fixedLiteralEncoding {
w.writeFixedHeader(eof)
} else {
w.writeDynamicHeader(numLiterals, numOffsets, numCodegens, eof)
}
leCodes := literalEncoding.codes
oeCodes := offsetEncoding.codes
for _, t := range tokens {
if t < matchType {
w.writeCode(leCodes[t.literal()])
} else {
// Write the length
length := t.length()
lengthCode := lengthCode(length)
w.writeCode(leCodes[lengthCode+lengthCodesStart])
extraLengthBits := uint(lengthExtraBits[lengthCode])
if extraLengthBits > 0 {
extraLength := int32(length - lengthBase[lengthCode])
w.writeBits(extraLength, extraLengthBits)
}
// Write the offset
offset := t.offset()
offsetCode := offsetCode(offset)
w.writeCode(oeCodes[offsetCode])
extraOffsetBits := uint(offsetExtraBits[offsetCode])
if extraOffsetBits > 0 {
extraOffset := int32(offset - offsetBase[offsetCode])
w.writeBits(extraOffset, extraOffsetBits)
}
}
}
}
// writeBlockDynamic will write a block as dynamic Huffman table
// compressed. This should be used, if the caller has a reasonable expectation
// that this block contains compressible data.
func (w *huffmanBitWriter) writeBlockDynamic(tok tokens, eof bool, input []byte) {
if w.err != nil {
return
}
for i := range w.literalFreq {
w.literalFreq[i] = 0
}
for i := range w.offsetFreq {
w.offsetFreq[i] = 0
}
tok.tokens[tok.n] = endBlockMarker
tokens := tok.tokens[0 : tok.n+1]
for _, t := range tokens {
if t < matchType {
w.literalFreq[t.literal()]++
} else {
length := t.length()
offset := t.offset()
w.literalFreq[lengthCodesStart+lengthCode(length)]++
w.offsetFreq[offsetCode(offset)]++
}
}
// get the number of literals
numLiterals := len(w.literalFreq)
for w.literalFreq[numLiterals-1] == 0 {
numLiterals--
}
// get the number of offsets
numOffsets := len(w.offsetFreq)
for numOffsets > 0 && w.offsetFreq[numOffsets-1] == 0 {
numOffsets--
}
if numOffsets == 0 {
// We haven't found a single match. If we want to go with the dynamic encoding,
// we should count at least one offset to be sure that the offset huffman tree could be encoded.
w.offsetFreq[0] = 1
numOffsets = 1
}
w.literalEncoding.generate(w.literalFreq, 15)
w.offsetEncoding.generate(w.offsetFreq, 15)
var numCodegens int
// Generate codegen and codegenFrequencies, which indicates how to encode
// the literalEncoding and the offsetEncoding.
w.generateCodegen(numLiterals, numOffsets, w.offsetEncoding)
w.codegenEncoding.generate(w.codegenFreq, 7)
numCodegens = len(w.codegenFreq)
for numCodegens > 4 && w.codegenFreq[codegenOrder[numCodegens-1]] == 0 {
numCodegens--
}
var literalEncoding = w.literalEncoding
var offsetEncoding = w.offsetEncoding
// Write Huffman table.
w.writeDynamicHeader(numLiterals, numOffsets, numCodegens, eof)
leCodes := literalEncoding.codes
oeCodes := offsetEncoding.codes
for _, t := range tokens {
if t < matchType {
w.writeCode(leCodes[t.literal()])
} else {
// Write the length
length := t.length()
lengthCode := lengthCode(length)
w.writeCode(leCodes[lengthCode+lengthCodesStart])
extraLengthBits := uint(lengthExtraBits[lengthCode])
if extraLengthBits > 0 {
extraLength := int32(length - lengthBase[lengthCode])
w.writeBits(extraLength, extraLengthBits)
}
// Write the offset
offset := t.offset()
offsetCode := offsetCode(offset)
w.writeCode(oeCodes[offsetCode])
extraOffsetBits := uint(offsetExtraBits[offsetCode])
if extraOffsetBits > 0 {
extraOffset := int32(offset - offsetBase[offsetCode])
w.writeBits(extraOffset, extraOffsetBits)
}
}
}
}
// static offset encoder used for huffman only encoding.
var huffOffset *huffmanEncoder
func init() {
var w = newHuffmanBitWriter(nil)
w.offsetFreq[0] = 1
huffOffset = newHuffmanEncoder(offsetCodeCount)
huffOffset.generate(w.offsetFreq, 15)
}
// writeBlockHuff will write a block of bytes as either
// Huffman encoded literals or uncompressed bytes if the
// results only gains very little from compression.
func (w *huffmanBitWriter) writeBlockHuff(eof bool, input []byte) {
if w.err != nil {
return
}
// Clear histogram
for i := range w.literalFreq {
w.literalFreq[i] = 0
}
// Add everything as literals
histogram(input, w.literalFreq)
w.literalFreq[endBlockMarker] = 1
const numLiterals = endBlockMarker + 1
const numOffsets = 1
w.literalEncoding.generate(w.literalFreq, 15)
// Figure out smallest code.
// Always use dynamic Huffman or Store
var numCodegens int
// Generate codegen and codegenFrequencies, which indicates how to encode
// the literalEncoding and the offsetEncoding.
w.generateCodegen(numLiterals, numOffsets, huffOffset)
w.codegenEncoding.generate(w.codegenFreq, 7)
numCodegens = len(w.codegenFreq)
for numCodegens > 4 && w.codegenFreq[codegenOrder[numCodegens-1]] == 0 {
numCodegens--
}
headerSize := int64(3+5+5+4+(3*numCodegens)) +
w.codegenEncoding.bitLength(w.codegenFreq) +
int64(w.codegenFreq[16]*2) +
int64(w.codegenFreq[17]*3) +
int64(w.codegenFreq[18]*7)
// Includes EOB marker
size := headerSize + w.literalEncoding.bitLength(w.literalFreq)
// Calculate stored size
var storedSize int64 = math.MaxInt64
var storedBytes = len(input)
if storedBytes <= maxStoreBlockSize {
storedSize = int64(storedBytes+5) * 8
}
// Store bytes, if we don't get a reasonable improvement.
if storedSize < (size + size>>4) {
w.writeStoredHeader(storedBytes, eof)
w.writeBytes(input)
return
}
// Huffman.
w.writeDynamicHeader(numLiterals, numOffsets, numCodegens, eof)
encoding := w.literalEncoding.codes
for _, t := range input {
// Bitwriting inlined, ~30% speedup
c := encoding[t]
w.bits |= uint64(c.code()) << w.nbits
w.nbits += c.bits()
if w.nbits >= 48 {
bits := w.bits
w.bits >>= 48
w.nbits -= 48
n := w.nbytes
w.bytes[n] = byte(bits)
w.bytes[n+1] = byte(bits >> 8)
w.bytes[n+2] = byte(bits >> 16)
w.bytes[n+3] = byte(bits >> 24)
w.bytes[n+4] = byte(bits >> 32)
w.bytes[n+5] = byte(bits >> 40)
n += 6
if n >= bufferSize-8 {
_, w.err = w.w.Write(w.bytes[:bufferSize-8])
if w.err != nil {
return
}
w.nbytes = 0
} else {
w.nbytes = n
}
}
}
w.writeCode(encoding[endBlockMarker])
}

368
vendor/github.com/klauspost/compress/flate/huffman_bit_writer_test.go generated vendored Normal file
View File

File diff suppressed because one or more lines are too long

363
vendor/github.com/klauspost/compress/flate/huffman_code.go generated vendored Normal file
View File

@@ -0,0 +1,363 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
import (
"math"
"sort"
)
type hcode uint32
type huffmanEncoder struct {
codes []hcode
freqcache []literalNode
bitCount [17]int32
lns literalNodeSorter
lfs literalFreqSorter
}
type literalNode struct {
literal uint16
freq int32
}
// A levelInfo describes the state of the constructed tree for a given depth.
type levelInfo struct {
// Our level. for better printing
level int32
// The frequency of the last node at this level
lastFreq int32
// The frequency of the next character to add to this level
nextCharFreq int32
// The frequency of the next pair (from level below) to add to this level.
// Only valid if the "needed" value of the next lower level is 0.
nextPairFreq int32
// The number of chains remaining to generate for this level before moving
// up to the next level
needed int32
}
func (h hcode) codeBits() (code uint16, bits uint8) {
return uint16(h), uint8(h >> 16)
}
func (h *hcode) set(code uint16, bits uint8) {
*h = hcode(code) | hcode(uint32(bits)<<16)
}
func (h *hcode) setBits(bits uint8) {
*h = hcode(*h&0xffff) | hcode(uint32(bits)<<16)
}
func toCode(code uint16, bits uint8) hcode {
return hcode(code) | hcode(uint32(bits)<<16)
}
func (h hcode) code() (code uint16) {
return uint16(h)
}
func (h hcode) bits() (bits uint) {
return uint(h >> 16)
}
func maxNode() literalNode { return literalNode{math.MaxUint16, math.MaxInt32} }
func newHuffmanEncoder(size int) *huffmanEncoder {
return &huffmanEncoder{codes: make([]hcode, size), freqcache: nil}
}
// Generates a HuffmanCode corresponding to the fixed literal table
func generateFixedLiteralEncoding() *huffmanEncoder {
h := newHuffmanEncoder(maxNumLit)
codes := h.codes
var ch uint16
for ch = 0; ch < maxNumLit; ch++ {
var bits uint16
var size uint8
switch {
case ch < 144:
// size 8, 000110000 .. 10111111
bits = ch + 48
size = 8
break
case ch < 256:
// size 9, 110010000 .. 111111111
bits = ch + 400 - 144
size = 9
break
case ch < 280:
// size 7, 0000000 .. 0010111
bits = ch - 256
size = 7
break
default:
// size 8, 11000000 .. 11000111
bits = ch + 192 - 280
size = 8
}
codes[ch] = toCode(reverseBits(bits, size), size)
}
return h
}
func generateFixedOffsetEncoding() *huffmanEncoder {
h := newHuffmanEncoder(30)
codes := h.codes
for ch := uint16(0); ch < 30; ch++ {
codes[ch] = toCode(reverseBits(ch, 5), 5)
}
return h
}
var fixedLiteralEncoding *huffmanEncoder = generateFixedLiteralEncoding()
var fixedOffsetEncoding *huffmanEncoder = generateFixedOffsetEncoding()
func (h *huffmanEncoder) bitLength(freq []int32) int64 {
var total int64
for i, f := range freq {
if f != 0 {
total += int64(f) * int64(h.codes[i].bits())
}
}
return total
}
const maxBitsLimit = 16
// Return the number of literals assigned to each bit size in the Huffman encoding
//
// This method is only called when list.length >= 3
// The cases of 0, 1, and 2 literals are handled by special case code.
//
// list An array of the literals with non-zero frequencies
// and their associated frequencies. The array is in order of increasing
// frequency, and has as its last element a special element with frequency
// MaxInt32
// maxBits The maximum number of bits that should be used to encode any literal.
// Must be less than 16.
// return An integer array in which array[i] indicates the number of literals
// that should be encoded in i bits.
func (h *huffmanEncoder) bitCounts(list []literalNode, maxBits int32) []int32 {
if maxBits >= maxBitsLimit {
panic("flate: maxBits too large")
}
n := int32(len(list))
list = list[0 : n+1]
list[n] = maxNode()
// The tree can't have greater depth than n - 1, no matter what. This
// saves a little bit of work in some small cases
if maxBits > n-1 {
maxBits = n - 1
}
// Create information about each of the levels.
// A bogus "Level 0" whose sole purpose is so that
// level1.prev.needed==0. This makes level1.nextPairFreq
// be a legitimate value that never gets chosen.
var levels [maxBitsLimit]levelInfo
// leafCounts[i] counts the number of literals at the left
// of ancestors of the rightmost node at level i.
// leafCounts[i][j] is the number of literals at the left
// of the level j ancestor.
var leafCounts [maxBitsLimit][maxBitsLimit]int32
for level := int32(1); level <= maxBits; level++ {
// For every level, the first two items are the first two characters.
// We initialize the levels as if we had already figured this out.
levels[level] = levelInfo{
level: level,
lastFreq: list[1].freq,
nextCharFreq: list[2].freq,
nextPairFreq: list[0].freq + list[1].freq,
}
leafCounts[level][level] = 2
if level == 1 {
levels[level].nextPairFreq = math.MaxInt32
}
}
// We need a total of 2*n - 2 items at top level and have already generated 2.
levels[maxBits].needed = 2*n - 4
level := maxBits
for {
l := &levels[level]
if l.nextPairFreq == math.MaxInt32 && l.nextCharFreq == math.MaxInt32 {
// We've run out of both leafs and pairs.
// End all calculations for this level.
// To make sure we never come back to this level or any lower level,
// set nextPairFreq impossibly large.
l.needed = 0
levels[level+1].nextPairFreq = math.MaxInt32
level++
continue
}
prevFreq := l.lastFreq
if l.nextCharFreq < l.nextPairFreq {
// The next item on this row is a leaf node.
n := leafCounts[level][level] + 1
l.lastFreq = l.nextCharFreq
// Lower leafCounts are the same of the previous node.
leafCounts[level][level] = n
l.nextCharFreq = list[n].freq
} else {
// The next item on this row is a pair from the previous row.
// nextPairFreq isn't valid until we generate two
// more values in the level below
l.lastFreq = l.nextPairFreq
// Take leaf counts from the lower level, except counts[level] remains the same.
copy(leafCounts[level][:level], leafCounts[level-1][:level])
levels[l.level-1].needed = 2
}
if l.needed--; l.needed == 0 {
// We've done everything we need to do for this level.
// Continue calculating one level up. Fill in nextPairFreq
// of that level with the sum of the two nodes we've just calculated on
// this level.
if l.level == maxBits {
// All done!
break
}
levels[l.level+1].nextPairFreq = prevFreq + l.lastFreq
level++
} else {
// If we stole from below, move down temporarily to replenish it.
for levels[level-1].needed > 0 {
level--
}
}
}
// Somethings is wrong if at the end, the top level is null or hasn't used
// all of the leaves.
if leafCounts[maxBits][maxBits] != n {
panic("leafCounts[maxBits][maxBits] != n")
}
bitCount := h.bitCount[:maxBits+1]
//make([]int32, maxBits+1)
bits := 1
counts := &leafCounts[maxBits]
for level := maxBits; level > 0; level-- {
// chain.leafCount gives the number of literals requiring at least "bits"
// bits to encode.
bitCount[bits] = counts[level] - counts[level-1]
bits++
}
return bitCount
}
// Look at the leaves and assign them a bit count and an encoding as specified
// in RFC 1951 3.2.2
func (h *huffmanEncoder) assignEncodingAndSize(bitCount []int32, list []literalNode) {
code := uint16(0)
for n, bits := range bitCount {
code <<= 1
if n == 0 || bits == 0 {
continue
}
// The literals list[len(list)-bits] .. list[len(list)-bits]
// are encoded using "bits" bits, and get the values
// code, code + 1, .... The code values are
// assigned in literal order (not frequency order).
chunk := list[len(list)-int(bits):]
h.lns.Sort(chunk)
for _, node := range chunk {
h.codes[node.literal] = toCode(reverseBits(code, uint8(n)), uint8(n))
code++
}
list = list[0 : len(list)-int(bits)]
}
}
// Update this Huffman Code object to be the minimum code for the specified frequency count.
//
// freq An array of frequencies, in which frequency[i] gives the frequency of literal i.
// maxBits The maximum number of bits to use for any literal.
func (h *huffmanEncoder) generate(freq []int32, maxBits int32) {
if h.freqcache == nil {
h.freqcache = make([]literalNode, 300)
}
list := h.freqcache[:len(freq)+1]
// Number of non-zero literals
count := 0
// Set list to be the set of all non-zero literals and their frequencies
for i, f := range freq {
if f != 0 {
list[count] = literalNode{uint16(i), f}
count++
} else {
list[count] = literalNode{}
//h.codeBits[i] = 0
h.codes[i].setBits(0)
}
}
list[len(freq)] = literalNode{}
// If freq[] is shorter than codeBits[], fill rest of codeBits[] with zeros
// FIXME: Doesn't do what it says on the tin (klauspost)
//h.codeBits = h.codeBits[0:len(freq)]
list = list[0:count]
if count <= 2 {
// Handle the small cases here, because they are awkward for the general case code. With
// two or fewer literals, everything has bit length 1.
for i, node := range list {
// "list" is in order of increasing literal value.
h.codes[node.literal].set(uint16(i), 1)
//h.codeBits[node.literal] = 1
//h.code[node.literal] = uint16(i)
}
return
}
h.lfs.Sort(list)
// Get the number of literals for each bit count
bitCount := h.bitCounts(list, maxBits)
// And do the assignment
h.assignEncodingAndSize(bitCount, list)
}
type literalNodeSorter []literalNode
func (s *literalNodeSorter) Sort(a []literalNode) {
*s = literalNodeSorter(a)
sort.Sort(s)
}
func (s literalNodeSorter) Len() int { return len(s) }
func (s literalNodeSorter) Less(i, j int) bool {
return s[i].literal < s[j].literal
}
func (s literalNodeSorter) Swap(i, j int) { s[i], s[j] = s[j], s[i] }
type literalFreqSorter []literalNode
func (s *literalFreqSorter) Sort(a []literalNode) {
*s = literalFreqSorter(a)
sort.Sort(s)
}
func (s literalFreqSorter) Len() int { return len(s) }
func (s literalFreqSorter) Less(i, j int) bool {
if s[i].freq == s[j].freq {
return s[i].literal < s[j].literal
}
return s[i].freq < s[j].freq
}
func (s literalFreqSorter) Swap(i, j int) { s[i], s[j] = s[j], s[i] }

846
vendor/github.com/klauspost/compress/flate/inflate.go generated vendored Normal file
View File

@@ -0,0 +1,846 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
//go:generate go run gen.go -output fixedhuff.go
// Package flate implements the DEFLATE compressed data format, described in
// RFC 1951. The gzip and zlib packages implement access to DEFLATE-based file
// formats.
package flate
import (
"bufio"
"io"
"strconv"
)
const (
maxCodeLen = 16 // max length of Huffman code
maxHist = 32768 // max history required
// The next three numbers come from the RFC section 3.2.7, with the
// additional proviso in section 3.2.5 which implies that distance codes
// 30 and 31 should never occur in compressed data.
maxNumLit = 286
maxNumDist = 30
numCodes = 19 // number of codes in Huffman meta-code
)
// A CorruptInputError reports the presence of corrupt input at a given offset.
type CorruptInputError int64
func (e CorruptInputError) Error() string {
return "flate: corrupt input before offset " + strconv.FormatInt(int64(e), 10)
}
// An InternalError reports an error in the flate code itself.
type InternalError string
func (e InternalError) Error() string { return "flate: internal error: " + string(e) }
// A ReadError reports an error encountered while reading input.
type ReadError struct {
Offset int64 // byte offset where error occurred
Err error // error returned by underlying Read
}
func (e *ReadError) Error() string {
return "flate: read error at offset " + strconv.FormatInt(e.Offset, 10) + ": " + e.Err.Error()
}
// A WriteError reports an error encountered while writing output.
type WriteError struct {
Offset int64 // byte offset where error occurred
Err error // error returned by underlying Write
}
func (e *WriteError) Error() string {
return "flate: write error at offset " + strconv.FormatInt(e.Offset, 10) + ": " + e.Err.Error()
}
// Resetter resets a ReadCloser returned by NewReader or NewReaderDict to
// to switch to a new underlying Reader. This permits reusing a ReadCloser
// instead of allocating a new one.
type Resetter interface {
// Reset discards any buffered data and resets the Resetter as if it was
// newly initialized with the given reader.
Reset(r io.Reader, dict []byte) error
}
// Note that much of the implementation of huffmanDecoder is also copied
// into gen.go (in package main) for the purpose of precomputing the
// fixed huffman tables so they can be included statically.
// The data structure for decoding Huffman tables is based on that of
// zlib. There is a lookup table of a fixed bit width (huffmanChunkBits),
// For codes smaller than the table width, there are multiple entries
// (each combination of trailing bits has the same value). For codes
// larger than the table width, the table contains a link to an overflow
// table. The width of each entry in the link table is the maximum code
// size minus the chunk width.
// Note that you can do a lookup in the table even without all bits
// filled. Since the extra bits are zero, and the DEFLATE Huffman codes
// have the property that shorter codes come before longer ones, the
// bit length estimate in the result is a lower bound on the actual
// number of bits.
// chunk & 15 is number of bits
// chunk >> 4 is value, including table link
const (
huffmanChunkBits = 9
huffmanNumChunks = 1 << huffmanChunkBits
huffmanCountMask = 15
huffmanValueShift = 4
)
type huffmanDecoder struct {
min int // the minimum code length
chunks [huffmanNumChunks]uint32 // chunks as described above
links [][]uint32 // overflow links
linkMask uint32 // mask the width of the link table
}
// Initialize Huffman decoding tables from array of code lengths.
// Following this function, h is guaranteed to be initialized into a complete
// tree (i.e., neither over-subscribed nor under-subscribed). The exception is a
// degenerate case where the tree has only a single symbol with length 1. Empty
// trees are permitted.
func (h *huffmanDecoder) init(bits []int) bool {
// Sanity enables additional runtime tests during Huffman
// table construction. It's intended to be used during
// development to supplement the currently ad-hoc unit tests.
const sanity = false
if h.min != 0 {
*h = huffmanDecoder{}
}
// Count number of codes of each length,
// compute min and max length.
var count [maxCodeLen]int
var min, max int
for _, n := range bits {
if n == 0 {
continue
}
if min == 0 || n < min {
min = n
}
if n > max {
max = n
}
count[n]++
}
// Empty tree. The decompressor.huffSym function will fail later if the tree
// is used. Technically, an empty tree is only valid for the HDIST tree and
// not the HCLEN and HLIT tree. However, a stream with an empty HCLEN tree
// is guaranteed to fail since it will attempt to use the tree to decode the
// codes for the HLIT and HDIST trees. Similarly, an empty HLIT tree is
// guaranteed to fail later since the compressed data section must be
// composed of at least one symbol (the end-of-block marker).
if max == 0 {
return true
}
code := 0
var nextcode [maxCodeLen]int
for i := min; i <= max; i++ {
code <<= 1
nextcode[i] = code
code += count[i]
}
// Check that the coding is complete (i.e., that we've
// assigned all 2-to-the-max possible bit sequences).
// Exception: To be compatible with zlib, we also need to
// accept degenerate single-code codings. See also
// TestDegenerateHuffmanCoding.
if code != 1<<uint(max) && !(code == 1 && max == 1) {
return false
}
h.min = min
if max > huffmanChunkBits {
numLinks := 1 << (uint(max) - huffmanChunkBits)
h.linkMask = uint32(numLinks - 1)
// create link tables
link := nextcode[huffmanChunkBits+1] >> 1
h.links = make([][]uint32, huffmanNumChunks-link)
for j := uint(link); j < huffmanNumChunks; j++ {
reverse := int(reverseByte[j>>8]) | int(reverseByte[j&0xff])<<8
reverse >>= uint(16 - huffmanChunkBits)
off := j - uint(link)
if sanity && h.chunks[reverse] != 0 {
panic("impossible: overwriting existing chunk")
}
h.chunks[reverse] = uint32(off<<huffmanValueShift | (huffmanChunkBits + 1))
h.links[off] = make([]uint32, numLinks)
}
}
for i, n := range bits {
if n == 0 {
continue
}
code := nextcode[n]
nextcode[n]++
chunk := uint32(i<<huffmanValueShift | n)
reverse := int(reverseByte[code>>8]) | int(reverseByte[code&0xff])<<8
reverse >>= uint(16 - n)
if n <= huffmanChunkBits {
for off := reverse; off < len(h.chunks); off += 1 << uint(n) {
// We should never need to overwrite
// an existing chunk. Also, 0 is
// never a valid chunk, because the
// lower 4 "count" bits should be
// between 1 and 15.
if sanity && h.chunks[off] != 0 {
panic("impossible: overwriting existing chunk")
}
h.chunks[off] = chunk
}
} else {
j := reverse & (huffmanNumChunks - 1)
if sanity && h.chunks[j]&huffmanCountMask != huffmanChunkBits+1 {
// Longer codes should have been
// associated with a link table above.
panic("impossible: not an indirect chunk")
}
value := h.chunks[j] >> huffmanValueShift
linktab := h.links[value]
reverse >>= huffmanChunkBits
for off := reverse; off < len(linktab); off += 1 << uint(n-huffmanChunkBits) {
if sanity && linktab[off] != 0 {
panic("impossible: overwriting existing chunk")
}
linktab[off] = chunk
}
}
}
if sanity {
// Above we've sanity checked that we never overwrote
// an existing entry. Here we additionally check that
// we filled the tables completely.
for i, chunk := range h.chunks {
if chunk == 0 {
// As an exception, in the degenerate
// single-code case, we allow odd
// chunks to be missing.
if code == 1 && i%2 == 1 {
continue
}
panic("impossible: missing chunk")
}
}
for _, linktab := range h.links {
for _, chunk := range linktab {
if chunk == 0 {
panic("impossible: missing chunk")
}
}
}
}
return true
}
// The actual read interface needed by NewReader.
// If the passed in io.Reader does not also have ReadByte,
// the NewReader will introduce its own buffering.
type Reader interface {
io.Reader
io.ByteReader
}
// Decompress state.
type decompressor struct {
// Input source.
r Reader
roffset int64
woffset int64
// Input bits, in top of b.
b uint32
nb uint
// Huffman decoders for literal/length, distance.
h1, h2 huffmanDecoder
// Length arrays used to define Huffman codes.
bits *[maxNumLit + maxNumDist]int
codebits *[numCodes]int
// Output history, buffer.
hist *[maxHist]byte
hp int // current output position in buffer
hw int // have written hist[0:hw] already
hfull bool // buffer has filled at least once
// Temporary buffer (avoids repeated allocation).
buf [4]byte
// Next step in the decompression,
// and decompression state.
step func(*decompressor)
final bool
err error
toRead []byte
hl, hd *huffmanDecoder
copyLen int
copyDist int
}
func (f *decompressor) nextBlock() {
if f.final {
if f.hw != f.hp {
f.flush((*decompressor).nextBlock)
return
}
f.err = io.EOF
return
}
for f.nb < 1+2 {
if f.err = f.moreBits(); f.err != nil {
return
}
}
f.final = f.b&1 == 1
f.b >>= 1
typ := f.b & 3
f.b >>= 2
f.nb -= 1 + 2
switch typ {
case 0:
f.dataBlock()
case 1:
// compressed, fixed Huffman tables
f.hl = &fixedHuffmanDecoder
f.hd = nil
f.huffmanBlock()
case 2:
// compressed, dynamic Huffman tables
if f.err = f.readHuffman(); f.err != nil {
break
}
f.hl = &f.h1
f.hd = &f.h2
f.huffmanBlock()
default:
// 3 is reserved.
f.err = CorruptInputError(f.roffset)
}
}
func (f *decompressor) Read(b []byte) (int, error) {
for {
if len(f.toRead) > 0 {
n := copy(b, f.toRead)
f.toRead = f.toRead[n:]
return n, nil
}
if f.err != nil {
return 0, f.err
}
f.step(f)
}
}
// Support the io.WriteTo interface for io.Copy and friends.
func (f *decompressor) WriteTo(w io.Writer) (int64, error) {
total := int64(0)
for {
if f.err != nil {
if f.err == io.EOF {
return total, nil
}
return total, f.err
}
if len(f.toRead) > 0 {
var n int
n, f.err = w.Write(f.toRead)
if f.err != nil {
return total, f.err
}
if n != len(f.toRead) {
return total, io.ErrShortWrite
}
f.toRead = f.toRead[:0]
total += int64(n)
}
f.step(f)
}
}
func (f *decompressor) Close() error {
if f.err == io.EOF {
return nil
}
return f.err
}
// RFC 1951 section 3.2.7.
// Compression with dynamic Huffman codes
var codeOrder = [...]int{16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15}
func (f *decompressor) readHuffman() error {
// HLIT[5], HDIST[5], HCLEN[4].
for f.nb < 5+5+4 {
if err := f.moreBits(); err != nil {
return err
}
}
nlit := int(f.b&0x1F) + 257
if nlit > maxNumLit {
return CorruptInputError(f.roffset)
}
f.b >>= 5
ndist := int(f.b&0x1F) + 1
if ndist > maxNumDist {
return CorruptInputError(f.roffset)
}
f.b >>= 5
nclen := int(f.b&0xF) + 4
// numCodes is 19, so nclen is always valid.
f.b >>= 4
f.nb -= 5 + 5 + 4
// (HCLEN+4)*3 bits: code lengths in the magic codeOrder order.
for i := 0; i < nclen; i++ {
for f.nb < 3 {
if err := f.moreBits(); err != nil {
return err
}
}
f.codebits[codeOrder[i]] = int(f.b & 0x7)
f.b >>= 3
f.nb -= 3
}
for i := nclen; i < len(codeOrder); i++ {
f.codebits[codeOrder[i]] = 0
}
if !f.h1.init(f.codebits[0:]) {
return CorruptInputError(f.roffset)
}
// HLIT + 257 code lengths, HDIST + 1 code lengths,
// using the code length Huffman code.
for i, n := 0, nlit+ndist; i < n; {
x, err := f.huffSym(&f.h1)
if err != nil {
return err
}
if x < 16 {
// Actual length.
f.bits[i] = x
i++
continue
}
// Repeat previous length or zero.
var rep int
var nb uint
var b int
switch x {
default:
return InternalError("unexpected length code")
case 16:
rep = 3
nb = 2
if i == 0 {
return CorruptInputError(f.roffset)
}
b = f.bits[i-1]
case 17:
rep = 3
nb = 3
b = 0
case 18:
rep = 11
nb = 7
b = 0
}
for f.nb < nb {
if err := f.moreBits(); err != nil {
return err
}
}
rep += int(f.b & uint32(1<<nb-1))
f.b >>= nb
f.nb -= nb
if i+rep > n {
return CorruptInputError(f.roffset)
}
for j := 0; j < rep; j++ {
f.bits[i] = b
i++
}
}
if !f.h1.init(f.bits[0:nlit]) || !f.h2.init(f.bits[nlit:nlit+ndist]) {
return CorruptInputError(f.roffset)
}
// In order to preserve the property that we never read any extra bytes
// after the end of the DEFLATE stream, huffSym conservatively reads min
// bits at a time until it decodes the symbol. However, since every block
// must end with an EOB marker, we can use that as the minimum number of
// bits to read and guarantee we never read past the end of the stream.
if f.bits[endBlockMarker] > 0 {
f.h1.min = f.bits[endBlockMarker] // Length of EOB marker
}
return nil
}
// Decode a single Huffman block from f.
// hl and hd are the Huffman states for the lit/length values
// and the distance values, respectively. If hd == nil, using the
// fixed distance encoding associated with fixed Huffman blocks.
func (f *decompressor) huffmanBlock() {
for {
v, err := f.huffSym(f.hl)
if err != nil {
f.err = err
return
}
var n uint // number of bits extra
var length int
switch {
case v < 256:
f.hist[f.hp] = byte(v)
f.hp++
if f.hp == len(f.hist) {
// After the flush, continue this loop.
f.flush((*decompressor).huffmanBlock)
return
}
continue
case v == 256:
// Done with huffman block; read next block.
f.step = (*decompressor).nextBlock
return
// otherwise, reference to older data
case v < 265:
length = v - (257 - 3)
n = 0
case v < 269:
length = v*2 - (265*2 - 11)
n = 1
case v < 273:
length = v*4 - (269*4 - 19)
n = 2
case v < 277:
length = v*8 - (273*8 - 35)
n = 3
case v < 281:
length = v*16 - (277*16 - 67)
n = 4
case v < 285:
length = v*32 - (281*32 - 131)
n = 5
case v < maxNumLit:
length = 258
n = 0
default:
f.err = CorruptInputError(f.roffset)
return
}
if n > 0 {
for f.nb < n {
if err = f.moreBits(); err != nil {
f.err = err
return
}
}
length += int(f.b & uint32(1<<n-1))
f.b >>= n
f.nb -= n
}
var dist int
if f.hd == nil {
for f.nb < 5 {
if err = f.moreBits(); err != nil {
f.err = err
return
}
}
dist = int(reverseByte[(f.b&0x1F)<<3])
f.b >>= 5
f.nb -= 5
} else {
if dist, err = f.huffSym(f.hd); err != nil {
f.err = err
return
}
}
switch {
case dist < 4:
dist++
case dist < maxNumDist:
nb := uint(dist-2) >> 1
// have 1 bit in bottom of dist, need nb more.
extra := (dist & 1) << nb
for f.nb < nb {
if err = f.moreBits(); err != nil {
f.err = err
return
}
}
extra |= int(f.b & uint32(1<<nb-1))
f.b >>= nb
f.nb -= nb
dist = 1<<(nb+1) + 1 + extra
default:
f.err = CorruptInputError(f.roffset)
return
}
// Copy history[-dist:-dist+length] into output.
if dist > len(f.hist) {
f.err = InternalError("bad history distance")
return
}
// No check on length; encoding can be prescient.
if !f.hfull && dist > f.hp {
f.err = CorruptInputError(f.roffset)
return
}
f.copyLen, f.copyDist = length, dist
if f.copyHist() {
return
}
}
}
// copyHist copies f.copyLen bytes from f.hist (f.copyDist bytes ago) to itself.
// It reports whether the f.hist buffer is full.
func (f *decompressor) copyHist() bool {
p := f.hp - f.copyDist
if p < 0 {
p += len(f.hist)
}
for f.copyLen > 0 {
n := f.copyLen
if x := len(f.hist) - f.hp; n > x {
n = x
}
if x := len(f.hist) - p; n > x {
n = x
}
forwardCopy(f.hist[:], f.hp, p, n)
p += n
f.hp += n
f.copyLen -= n
if f.hp == len(f.hist) {
// After flush continue copying out of history.
f.flush((*decompressor).copyHuff)
return true
}
if p == len(f.hist) {
p = 0
}
}
return false
}
func (f *decompressor) copyHuff() {
if f.copyHist() {
return
}
f.huffmanBlock()
}
// Copy a single uncompressed data block from input to output.
func (f *decompressor) dataBlock() {
// Uncompressed.
// Discard current half-byte.
f.nb = 0
f.b = 0
// Length then ones-complement of length.
nr, err := io.ReadFull(f.r, f.buf[0:4])
f.roffset += int64(nr)
if err != nil {
f.err = &ReadError{f.roffset, err}
return
}
n := int(f.buf[0]) | int(f.buf[1])<<8
nn := int(f.buf[2]) | int(f.buf[3])<<8
if uint16(nn) != uint16(^n) {
f.err = CorruptInputError(f.roffset)
return
}
if n == 0 {
// 0-length block means sync
f.flush((*decompressor).nextBlock)
return
}
f.copyLen = n
f.copyData()
}
// copyData copies f.copyLen bytes from the underlying reader into f.hist.
// It pauses for reads when f.hist is full.
func (f *decompressor) copyData() {
n := f.copyLen
for n > 0 {
m := len(f.hist) - f.hp
if m > n {
m = n
}
m, err := io.ReadFull(f.r, f.hist[f.hp:f.hp+m])
f.roffset += int64(m)
if err != nil {
f.err = &ReadError{f.roffset, err}
return
}
n -= m
f.hp += m
if f.hp == len(f.hist) {
f.copyLen = n
f.flush((*decompressor).copyData)
return
}
}
f.step = (*decompressor).nextBlock
}
func (f *decompressor) setDict(dict []byte) {
if len(dict) > len(f.hist) {
// Will only remember the tail.
dict = dict[len(dict)-len(f.hist):]
}
f.hp = copy(f.hist[:], dict)
if f.hp == len(f.hist) {
f.hp = 0
f.hfull = true
}
f.hw = f.hp
}
func (f *decompressor) moreBits() error {
c, err := f.r.ReadByte()
if err != nil {
if err == io.EOF {
err = io.ErrUnexpectedEOF
}
return err
}
f.roffset++
f.b |= uint32(c) << f.nb
f.nb += 8
return nil
}
// Read the next Huffman-encoded symbol from f according to h.
func (f *decompressor) huffSym(h *huffmanDecoder) (int, error) {
// Since a huffmanDecoder can be empty or be composed of a degenerate tree
// with single element, huffSym must error on these two edge cases. In both
// cases, the chunks slice will be 0 for the invalid sequence, leading it
// satisfy the n == 0 check below.
n := uint(h.min)
for {
for f.nb < n {
if err := f.moreBits(); err != nil {
return 0, err
}
}
chunk := h.chunks[f.b&(huffmanNumChunks-1)]
n = uint(chunk & huffmanCountMask)
if n > huffmanChunkBits {
chunk = h.links[chunk>>huffmanValueShift][(f.b>>huffmanChunkBits)&h.linkMask]
n = uint(chunk & huffmanCountMask)
}
if n <= f.nb {
if n == 0 {
f.err = CorruptInputError(f.roffset)
return 0, f.err
}
f.b >>= n
f.nb -= n
return int(chunk >> huffmanValueShift), nil
}
}
}
// Flush any buffered output to the underlying writer.
func (f *decompressor) flush(step func(*decompressor)) {
f.toRead = f.hist[f.hw:f.hp]
f.woffset += int64(f.hp - f.hw)
f.hw = f.hp
if f.hp == len(f.hist) {
f.hp = 0
f.hw = 0
f.hfull = true
}
f.step = step
}
func makeReader(r io.Reader) Reader {
if rr, ok := r.(Reader); ok {
return rr
}
return bufio.NewReader(r)
}
func (f *decompressor) Reset(r io.Reader, dict []byte) error {
*f = decompressor{
r: makeReader(r),
bits: f.bits,
codebits: f.codebits,
hist: f.hist,
step: (*decompressor).nextBlock,
}
if dict != nil {
f.setDict(dict)
}
return nil
}
// NewReader returns a new ReadCloser that can be used
// to read the uncompressed version of r.
// If r does not also implement io.ByteReader,
// the decompressor may read more data than necessary from r.
// It is the caller's responsibility to call Close on the ReadCloser
// when finished reading.
//
// The ReadCloser returned by NewReader also implements Resetter.
func NewReader(r io.Reader) io.ReadCloser {
var f decompressor
f.bits = new([maxNumLit + maxNumDist]int)
f.codebits = new([numCodes]int)
f.r = makeReader(r)
f.hist = new([maxHist]byte)
f.step = (*decompressor).nextBlock
return &f
}
// NewReaderDict is like NewReader but initializes the reader
// with a preset dictionary. The returned Reader behaves as if
// the uncompressed data stream started with the given dictionary,
// which has already been read. NewReaderDict is typically used
// to read data compressed by NewWriterDict.
//
// The ReadCloser returned by NewReader also implements Resetter.
func NewReaderDict(r io.Reader, dict []byte) io.ReadCloser {
var f decompressor
f.r = makeReader(r)
f.hist = new([maxHist]byte)
f.bits = new([maxNumLit + maxNumDist]int)
f.codebits = new([numCodes]int)
f.step = (*decompressor).nextBlock
f.setDict(dict)
return &f
}

225
vendor/github.com/klauspost/compress/flate/inflate_test.go generated vendored Normal file
View File

@@ -0,0 +1,225 @@
// Copyright 2014 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
import (
"bytes"
"crypto/rand"
"io"
"io/ioutil"
"strconv"
"strings"
"testing"
)
func TestReset(t *testing.T) {
ss := []string{
"lorem ipsum izzle fo rizzle",
"the quick brown fox jumped over",
}
deflated := make([]bytes.Buffer, 2)
for i, s := range ss {
w, _ := NewWriter(&deflated[i], 1)
w.Write([]byte(s))
w.Close()
}
inflated := make([]bytes.Buffer, 2)
f := NewReader(&deflated[0])
io.Copy(&inflated[0], f)
f.(Resetter).Reset(&deflated[1], nil)
io.Copy(&inflated[1], f)
f.Close()
for i, s := range ss {
if s != inflated[i].String() {
t.Errorf("inflated[%d]:\ngot %q\nwant %q", i, inflated[i], s)
}
}
}
// Tests ported from zlib/test/infcover.c
type infTest struct {
hex string
id string
n int
}
var infTests = []infTest{
infTest{"0 0 0 0 0", "invalid stored block lengths", 1},
infTest{"3 0", "fixed", 0},
infTest{"6", "invalid block type", 1},
infTest{"1 1 0 fe ff 0", "stored", 0},
infTest{"fc 0 0", "too many length or distance symbols", 1},
infTest{"4 0 fe ff", "invalid code lengths set", 1},
infTest{"4 0 24 49 0", "invalid bit length repeat", 1},
infTest{"4 0 24 e9 ff ff", "invalid bit length repeat", 1},
infTest{"4 0 24 e9 ff 6d", "invalid code -- missing end-of-block", 1},
infTest{"4 80 49 92 24 49 92 24 71 ff ff 93 11 0", "invalid literal/lengths set", 1},
infTest{"4 80 49 92 24 49 92 24 f b4 ff ff c3 84", "invalid distances set", 1},
infTest{"4 c0 81 8 0 0 0 0 20 7f eb b 0 0", "invalid literal/length code", 1},
infTest{"2 7e ff ff", "invalid distance code", 1},
infTest{"c c0 81 0 0 0 0 0 90 ff 6b 4 0", "invalid distance too far back", 1},
// also trailer mismatch just in inflate()
infTest{"1f 8b 8 0 0 0 0 0 0 0 3 0 0 0 0 1", "incorrect data check", -1},
infTest{"1f 8b 8 0 0 0 0 0 0 0 3 0 0 0 0 0 0 0 0 1", "incorrect length check", -1},
infTest{"5 c0 21 d 0 0 0 80 b0 fe 6d 2f 91 6c", "pull 17", 0},
infTest{"5 e0 81 91 24 cb b2 2c 49 e2 f 2e 8b 9a 47 56 9f fb fe ec d2 ff 1f", "long code", 0},
infTest{"ed c0 1 1 0 0 0 40 20 ff 57 1b 42 2c 4f", "length extra", 0},
infTest{"ed cf c1 b1 2c 47 10 c4 30 fa 6f 35 1d 1 82 59 3d fb be 2e 2a fc f c", "long distance and extra", 0},
infTest{"ed c0 81 0 0 0 0 80 a0 fd a9 17 a9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 6", "window end", 0},
}
func TestInflate(t *testing.T) {
for _, test := range infTests {
hex := strings.Split(test.hex, " ")
data := make([]byte, len(hex))
for i, h := range hex {
b, _ := strconv.ParseInt(h, 16, 32)
data[i] = byte(b)
}
buf := bytes.NewReader(data)
r := NewReader(buf)
_, err := io.Copy(ioutil.Discard, r)
if (test.n == 0 && err == nil) || (test.n != 0 && err != nil) {
t.Logf("%q: OK:", test.id)
t.Logf(" - got %v", err)
continue
}
if test.n == 0 && err != nil {
t.Errorf("%q: Expected no error, but got %v", test.id, err)
continue
}
if test.n != 0 && err == nil {
t.Errorf("%q:Expected an error, but got none", test.id)
continue
}
t.Fatal(test.n, err)
}
for _, test := range infOutTests {
hex := strings.Split(test.hex, " ")
data := make([]byte, len(hex))
for i, h := range hex {
b, _ := strconv.ParseInt(h, 16, 32)
data[i] = byte(b)
}
buf := bytes.NewReader(data)
r := NewReader(buf)
_, err := io.Copy(ioutil.Discard, r)
if test.err == (err != nil) {
t.Logf("%q: OK:", test.id)
t.Logf(" - got %v", err)
continue
}
if test.err == false && err != nil {
t.Errorf("%q: Expected no error, but got %v", test.id, err)
continue
}
if test.err && err == nil {
t.Errorf("%q: Expected an error, but got none", test.id)
continue
}
t.Fatal(test.err, err)
}
}
// Tests ported from zlib/test/infcover.c
// Since zlib inflate is push (writer) instead of pull (reader)
// some of the window size tests have been removed, since they
// are irrelevant.
type infOutTest struct {
hex string
id string
step int
win int
length int
err bool
}
var infOutTests = []infOutTest{
infOutTest{"2 8 20 80 0 3 0", "inflate_fast TYPE return", 0, -15, 258, false},
infOutTest{"63 18 5 40 c 0", "window wrap", 3, -8, 300, false},
infOutTest{"e5 e0 81 ad 6d cb b2 2c c9 01 1e 59 63 ae 7d ee fb 4d fd b5 35 41 68 ff 7f 0f 0 0 0", "fast length extra bits", 0, -8, 258, true},
infOutTest{"25 fd 81 b5 6d 59 b6 6a 49 ea af 35 6 34 eb 8c b9 f6 b9 1e ef 67 49 50 fe ff ff 3f 0 0", "fast distance extra bits", 0, -8, 258, true},
infOutTest{"3 7e 0 0 0 0 0", "fast invalid distance code", 0, -8, 258, true},
infOutTest{"1b 7 0 0 0 0 0", "fast invalid literal/length code", 0, -8, 258, true},
infOutTest{"d c7 1 ae eb 38 c 4 41 a0 87 72 de df fb 1f b8 36 b1 38 5d ff ff 0", "fast 2nd level codes and too far back", 0, -8, 258, true},
infOutTest{"63 18 5 8c 10 8 0 0 0 0", "very common case", 0, -8, 259, false},
infOutTest{"63 60 60 18 c9 0 8 18 18 18 26 c0 28 0 29 0 0 0", "contiguous and wrap around window", 6, -8, 259, false},
infOutTest{"63 0 3 0 0 0 0 0", "copy direct from output", 0, -8, 259, false},
infOutTest{"1f 8b 0 0", "bad gzip method", 0, 31, 0, true},
infOutTest{"1f 8b 8 80", "bad gzip flags", 0, 31, 0, true},
infOutTest{"77 85", "bad zlib method", 0, 15, 0, true},
infOutTest{"78 9c", "bad zlib window size", 0, 8, 0, true},
infOutTest{"1f 8b 8 1e 0 0 0 0 0 0 1 0 0 0 0 0 0", "bad header crc", 0, 47, 1, true},
infOutTest{"1f 8b 8 2 0 0 0 0 0 0 1d 26 3 0 0 0 0 0 0 0 0 0", "check gzip length", 0, 47, 0, true},
infOutTest{"78 90", "bad zlib header check", 0, 47, 0, true},
infOutTest{"8 b8 0 0 0 1", "need dictionary", 0, 8, 0, true},
infOutTest{"63 18 68 30 d0 0 0", "force split window update", 4, -8, 259, false},
infOutTest{"3 0", "use fixed blocks", 0, -15, 1, false},
infOutTest{"", "bad window size", 0, 1, 0, true},
}
func TestWriteTo(t *testing.T) {
input := make([]byte, 100000)
n, err := rand.Read(input)
if err != nil {
t.Fatal(err)
}
if n != len(input) {
t.Fatal("did not fill buffer")
}
compressed := &bytes.Buffer{}
w, err := NewWriter(compressed, -2)
if err != nil {
t.Fatal(err)
}
n, err = w.Write(input)
if err != nil {
t.Fatal(err)
}
if n != len(input) {
t.Fatal("did not fill buffer")
}
w.Close()
buf := compressed.Bytes()
dec := NewReader(bytes.NewBuffer(buf))
// ReadAll does not use WriteTo, but we wrap it in a NopCloser to be sure.
readall, err := ioutil.ReadAll(ioutil.NopCloser(dec))
if err != nil {
t.Fatal(err)
}
if len(readall) != len(input) {
t.Fatal("did not decompress everything")
}
dec = NewReader(bytes.NewBuffer(buf))
wtbuf := &bytes.Buffer{}
written, err := dec.(io.WriterTo).WriteTo(wtbuf)
if err != nil {
t.Fatal(err)
}
if written != int64(len(input)) {
t.Error("Returned length did not match, expected", len(input), "got", written)
}
if wtbuf.Len() != len(input) {
t.Error("Actual Length did not match, expected", len(input), "got", wtbuf.Len())
}
if bytes.Compare(wtbuf.Bytes(), input) != 0 {
t.Fatal("output did not match input")
}
}

97
vendor/github.com/klauspost/compress/flate/reader_test.go generated vendored Normal file
View File

@@ -0,0 +1,97 @@
// Copyright 2012 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
import (
"bytes"
"io"
"io/ioutil"
"runtime"
"strings"
"testing"
)
func TestNlitOutOfRange(t *testing.T) {
// Trying to decode this bogus flate data, which has a Huffman table
// with nlit=288, should not panic.
io.Copy(ioutil.Discard, NewReader(strings.NewReader(
"\xfc\xfe\x36\xe7\x5e\x1c\xef\xb3\x55\x58\x77\xb6\x56\xb5\x43\xf4"+
"\x6f\xf2\xd2\xe6\x3d\x99\xa0\x85\x8c\x48\xeb\xf8\xda\x83\x04\x2a"+
"\x75\xc4\xf8\x0f\x12\x11\xb9\xb4\x4b\x09\xa0\xbe\x8b\x91\x4c")))
}
const (
digits = iota
twain
)
var testfiles = []string{
// Digits is the digits of the irrational number e. Its decimal representation
// does not repeat, but there are only 10 possible digits, so it should be
// reasonably compressible.
digits: "../testdata/e.txt",
// Twain is Project Gutenberg's edition of Mark Twain's classic English novel.
twain: "../testdata/Mark.Twain-Tom.Sawyer.txt",
}
func benchmarkDecode(b *testing.B, testfile, level, n int) {
b.ReportAllocs()
b.StopTimer()
b.SetBytes(int64(n))
buf0, err := ioutil.ReadFile(testfiles[testfile])
if err != nil {
b.Fatal(err)
}
if len(buf0) == 0 {
b.Fatalf("test file %q has no data", testfiles[testfile])
}
compressed := new(bytes.Buffer)
w, err := NewWriter(compressed, level)
if err != nil {
b.Fatal(err)
}
for i := 0; i < n; i += len(buf0) {
if len(buf0) > n-i {
buf0 = buf0[:n-i]
}
io.Copy(w, bytes.NewReader(buf0))
}
w.Close()
buf1 := compressed.Bytes()
buf0, compressed, w = nil, nil, nil
runtime.GC()
b.StartTimer()
for i := 0; i < b.N; i++ {
io.Copy(ioutil.Discard, NewReader(bytes.NewReader(buf1)))
}
}
// These short names are so that gofmt doesn't break the BenchmarkXxx function
// bodies below over multiple lines.
const (
constant = ConstantCompression
speed = BestSpeed
default_ = DefaultCompression
compress = BestCompression
)
func BenchmarkDecodeDigitsSpeed1e4(b *testing.B) { benchmarkDecode(b, digits, speed, 1e4) }
func BenchmarkDecodeDigitsSpeed1e5(b *testing.B) { benchmarkDecode(b, digits, speed, 1e5) }
func BenchmarkDecodeDigitsSpeed1e6(b *testing.B) { benchmarkDecode(b, digits, speed, 1e6) }
func BenchmarkDecodeDigitsDefault1e4(b *testing.B) { benchmarkDecode(b, digits, default_, 1e4) }
func BenchmarkDecodeDigitsDefault1e5(b *testing.B) { benchmarkDecode(b, digits, default_, 1e5) }
func BenchmarkDecodeDigitsDefault1e6(b *testing.B) { benchmarkDecode(b, digits, default_, 1e6) }
func BenchmarkDecodeDigitsCompress1e4(b *testing.B) { benchmarkDecode(b, digits, compress, 1e4) }
func BenchmarkDecodeDigitsCompress1e5(b *testing.B) { benchmarkDecode(b, digits, compress, 1e5) }
func BenchmarkDecodeDigitsCompress1e6(b *testing.B) { benchmarkDecode(b, digits, compress, 1e6) }
func BenchmarkDecodeTwainSpeed1e4(b *testing.B) { benchmarkDecode(b, twain, speed, 1e4) }
func BenchmarkDecodeTwainSpeed1e5(b *testing.B) { benchmarkDecode(b, twain, speed, 1e5) }
func BenchmarkDecodeTwainSpeed1e6(b *testing.B) { benchmarkDecode(b, twain, speed, 1e6) }
func BenchmarkDecodeTwainDefault1e4(b *testing.B) { benchmarkDecode(b, twain, default_, 1e4) }
func BenchmarkDecodeTwainDefault1e5(b *testing.B) { benchmarkDecode(b, twain, default_, 1e5) }
func BenchmarkDecodeTwainDefault1e6(b *testing.B) { benchmarkDecode(b, twain, default_, 1e6) }
func BenchmarkDecodeTwainCompress1e4(b *testing.B) { benchmarkDecode(b, twain, compress, 1e4) }
func BenchmarkDecodeTwainCompress1e5(b *testing.B) { benchmarkDecode(b, twain, compress, 1e5) }
func BenchmarkDecodeTwainCompress1e6(b *testing.B) { benchmarkDecode(b, twain, compress, 1e6) }

48
vendor/github.com/klauspost/compress/flate/reverse_bits.go generated vendored Normal file
View File

@@ -0,0 +1,48 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
var reverseByte = [256]byte{
0x00, 0x80, 0x40, 0xc0, 0x20, 0xa0, 0x60, 0xe0,
0x10, 0x90, 0x50, 0xd0, 0x30, 0xb0, 0x70, 0xf0,
0x08, 0x88, 0x48, 0xc8, 0x28, 0xa8, 0x68, 0xe8,
0x18, 0x98, 0x58, 0xd8, 0x38, 0xb8, 0x78, 0xf8,
0x04, 0x84, 0x44, 0xc4, 0x24, 0xa4, 0x64, 0xe4,
0x14, 0x94, 0x54, 0xd4, 0x34, 0xb4, 0x74, 0xf4,
0x0c, 0x8c, 0x4c, 0xcc, 0x2c, 0xac, 0x6c, 0xec,
0x1c, 0x9c, 0x5c, 0xdc, 0x3c, 0xbc, 0x7c, 0xfc,
0x02, 0x82, 0x42, 0xc2, 0x22, 0xa2, 0x62, 0xe2,
0x12, 0x92, 0x52, 0xd2, 0x32, 0xb2, 0x72, 0xf2,
0x0a, 0x8a, 0x4a, 0xca, 0x2a, 0xaa, 0x6a, 0xea,
0x1a, 0x9a, 0x5a, 0xda, 0x3a, 0xba, 0x7a, 0xfa,
0x06, 0x86, 0x46, 0xc6, 0x26, 0xa6, 0x66, 0xe6,
0x16, 0x96, 0x56, 0xd6, 0x36, 0xb6, 0x76, 0xf6,
0x0e, 0x8e, 0x4e, 0xce, 0x2e, 0xae, 0x6e, 0xee,
0x1e, 0x9e, 0x5e, 0xde, 0x3e, 0xbe, 0x7e, 0xfe,
0x01, 0x81, 0x41, 0xc1, 0x21, 0xa1, 0x61, 0xe1,
0x11, 0x91, 0x51, 0xd1, 0x31, 0xb1, 0x71, 0xf1,
0x09, 0x89, 0x49, 0xc9, 0x29, 0xa9, 0x69, 0xe9,
0x19, 0x99, 0x59, 0xd9, 0x39, 0xb9, 0x79, 0xf9,
0x05, 0x85, 0x45, 0xc5, 0x25, 0xa5, 0x65, 0xe5,
0x15, 0x95, 0x55, 0xd5, 0x35, 0xb5, 0x75, 0xf5,
0x0d, 0x8d, 0x4d, 0xcd, 0x2d, 0xad, 0x6d, 0xed,
0x1d, 0x9d, 0x5d, 0xdd, 0x3d, 0xbd, 0x7d, 0xfd,
0x03, 0x83, 0x43, 0xc3, 0x23, 0xa3, 0x63, 0xe3,
0x13, 0x93, 0x53, 0xd3, 0x33, 0xb3, 0x73, 0xf3,
0x0b, 0x8b, 0x4b, 0xcb, 0x2b, 0xab, 0x6b, 0xeb,
0x1b, 0x9b, 0x5b, 0xdb, 0x3b, 0xbb, 0x7b, 0xfb,
0x07, 0x87, 0x47, 0xc7, 0x27, 0xa7, 0x67, 0xe7,
0x17, 0x97, 0x57, 0xd7, 0x37, 0xb7, 0x77, 0xf7,
0x0f, 0x8f, 0x4f, 0xcf, 0x2f, 0xaf, 0x6f, 0xef,
0x1f, 0x9f, 0x5f, 0xdf, 0x3f, 0xbf, 0x7f, 0xff,
}
func reverseUint16(v uint16) uint16 {
return uint16(reverseByte[v>>8]) | uint16(reverseByte[v&0xFF])<<8
}
func reverseBits(number uint16, bitLength byte) uint16 {
return reverseUint16(number << uint8(16-bitLength))
}

558
vendor/github.com/klauspost/compress/flate/snappy.go generated vendored Normal file
View File

@@ -0,0 +1,558 @@
// Copyright 2011 The Snappy-Go Authors. All rights reserved.
// Modified for deflate by Klaus Post (c) 2015.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
// We limit how far copy back-references can go, the same as the C++ code.
const maxOffset = 1 << 15
// emitLiteral writes a literal chunk and returns the number of bytes written.
func emitLiteral(dst *tokens, lit []byte) {
ol := dst.n
for i, v := range lit {
dst.tokens[i+ol] = token(v)
}
dst.n += len(lit)
}
// emitCopy writes a copy chunk and returns the number of bytes written.
func emitCopy(dst *tokens, offset, length int) {
dst.tokens[dst.n] = matchToken(uint32(length-3), uint32(offset-minOffsetSize))
dst.n++
}
type snappyEnc interface {
Encode(dst *tokens, src []byte)
Reset()
}
func newSnappy(level int) snappyEnc {
if useSSE42 {
e := &snappySSE4{snappyGen: snappyGen{cur: 1}}
switch level {
case 3:
e.enc = e.encodeL3
return e
}
}
e := &snappyGen{cur: 1}
switch level {
case 1:
e.enc = e.encodeL1
case 2:
e.enc = e.encodeL2
case 3:
e.enc = e.encodeL3
default:
panic("invalid level specified")
}
return e
}
const tableBits = 14 // Bits used in the table
const tableSize = 1 << tableBits // Size of the table
// snappyGen maintains the table for matches,
// and the previous byte block for level 2.
// This is the generic implementation.
type snappyGen struct {
table [tableSize]int64
block [maxStoreBlockSize]byte
prev []byte
cur int
enc func(dst *tokens, src []byte)
}
func (e *snappyGen) Encode(dst *tokens, src []byte) {
e.enc(dst, src)
}
// EncodeL1 uses Snappy-like compression, but stores as Huffman
// blocks.
func (e *snappyGen) encodeL1(dst *tokens, src []byte) {
// Return early if src is short.
if len(src) <= 4 {
if len(src) != 0 {
emitLiteral(dst, src)
}
e.cur += 4
return
}
// Ensure that e.cur doesn't wrap, mainly an issue on 32 bits.
if e.cur > 1<<30 {
e.cur = 1
}
// Iterate over the source bytes.
var (
s int // The iterator position.
t int // The last position with the same hash as s.
lit int // The start position of any pending literal bytes.
)
for s+3 < len(src) {
// Update the hash table.
b0, b1, b2, b3 := src[s], src[s+1], src[s+2], src[s+3]
h := uint32(b0) | uint32(b1)<<8 | uint32(b2)<<16 | uint32(b3)<<24
p := &e.table[(h*0x1e35a7bd)>>(32-tableBits)]
// We need to to store values in [-1, inf) in table.
// To save some initialization time, we make sure that
// e.cur is never zero.
t, *p = int(*p)-e.cur, int64(s+e.cur)
offset := uint(s - t - 1)
// If t is invalid or src[s:s+4] differs from src[t:t+4], accumulate a literal byte.
if t < 0 || offset >= (maxOffset-1) || b0 != src[t] || b1 != src[t+1] || b2 != src[t+2] || b3 != src[t+3] {
// Skip 1 byte for 16 consecutive missed.
s += 1 + ((s - lit) >> 4)
continue
}
// Otherwise, we have a match. First, emit any pending literal bytes.
if lit != s {
emitLiteral(dst, src[lit:s])
}
// Extend the match to be as long as possible.
s0 := s
s1 := s + maxMatchLength
if s1 > len(src) {
s1 = len(src)
}
s, t = s+4, t+4
for s < s1 && src[s] == src[t] {
s++
t++
}
// Emit the copied bytes.
// inlined: emitCopy(dst, s-t, s-s0)
dst.tokens[dst.n] = matchToken(uint32(s-s0-3), uint32(s-t-minOffsetSize))
dst.n++
lit = s
}
// Emit any final pending literal bytes and return.
if lit != len(src) {
emitLiteral(dst, src[lit:])
}
e.cur += len(src)
}
// EncodeL2 uses a similar algorithm to level 1, but is capable
// of matching across blocks giving better compression at a small slowdown.
func (e *snappyGen) encodeL2(dst *tokens, src []byte) {
// Return early if src is short.
if len(src) <= 4 {
if len(src) != 0 {
emitLiteral(dst, src)
}
e.prev = nil
e.cur += len(src)
return
}
// Ensure that e.cur doesn't wrap, mainly an issue on 32 bits.
if e.cur > 1<<30 {
e.cur = 1
}
// Iterate over the source bytes.
var (
s int // The iterator position.
t int // The last position with the same hash as s.
lit int // The start position of any pending literal bytes.
)
for s+3 < len(src) {
// Update the hash table.
b0, b1, b2, b3 := src[s], src[s+1], src[s+2], src[s+3]
h := uint32(b0) | uint32(b1)<<8 | uint32(b2)<<16 | uint32(b3)<<24
p := &e.table[(h*0x1e35a7bd)>>(32-tableBits)]
// We need to to store values in [-1, inf) in table.
// To save some initialization time, we make sure that
// e.cur is never zero.
t, *p = int(*p)-e.cur, int64(s+e.cur)
// If t is positive, the match starts in the current block
if t >= 0 {
offset := uint(s - t - 1)
// Check that the offset is valid and that we match at least 4 bytes
if offset >= (maxOffset-1) || b0 != src[t] || b1 != src[t+1] || b2 != src[t+2] || b3 != src[t+3] {
// Skip 1 byte for 32 consecutive missed.
s += 1 + ((s - lit) >> 5)
continue
}
// Otherwise, we have a match. First, emit any pending literal bytes.
if lit != s {
emitLiteral(dst, src[lit:s])
}
// Extend the match to be as long as possible.
s0 := s
s1 := s + maxMatchLength
if s1 > len(src) {
s1 = len(src)
}
s, t = s+4, t+4
for s < s1 && src[s] == src[t] {
s++
t++
}
// Emit the copied bytes.
// inlined: emitCopy(dst, s-t, s-s0)
dst.tokens[dst.n] = matchToken(uint32(s-s0-3), uint32(s-t-minOffsetSize))
dst.n++
lit = s
continue
}
// We found a match in the previous block.
tp := len(e.prev) + t
if tp < 0 || t > -5 || s-t >= maxOffset || b0 != e.prev[tp] || b1 != e.prev[tp+1] || b2 != e.prev[tp+2] || b3 != e.prev[tp+3] {
// Skip 1 byte for 32 consecutive missed.
s += 1 + ((s - lit) >> 5)
continue
}
// Otherwise, we have a match. First, emit any pending literal bytes.
if lit != s {
emitLiteral(dst, src[lit:s])
}
// Extend the match to be as long as possible.
s0 := s
s1 := s + maxMatchLength
if s1 > len(src) {
s1 = len(src)
}
s, tp = s+4, tp+4
for s < s1 && src[s] == e.prev[tp] {
s++
tp++
if tp == len(e.prev) {
t = 0
// continue in current buffer
for s < s1 && src[s] == src[t] {
s++
t++
}
goto l
}
}
l:
// Emit the copied bytes.
if t < 0 {
t = tp - len(e.prev)
}
dst.tokens[dst.n] = matchToken(uint32(s-s0-3), uint32(s-t-minOffsetSize))
dst.n++
lit = s
}
// Emit any final pending literal bytes and return.
if lit != len(src) {
emitLiteral(dst, src[lit:])
}
e.cur += len(src)
// Store this block, if it was full length.
if len(src) == maxStoreBlockSize {
copy(e.block[:], src)
e.prev = e.block[:len(src)]
} else {
e.prev = nil
}
}
// EncodeL3 uses a similar algorithm to level 2, but is capable
// will keep two matches per hash.
// Both hashes are checked if the first isn't ok, and the longest is selected.
func (e *snappyGen) encodeL3(dst *tokens, src []byte) {
// Return early if src is short.
if len(src) <= 4 {
if len(src) != 0 {
emitLiteral(dst, src)
}
e.prev = nil
e.cur += len(src)
return
}
// Ensure that e.cur doesn't wrap, mainly an issue on 32 bits.
if e.cur > 1<<30 {
e.cur = 1
}
// Iterate over the source bytes.
var (
s int // The iterator position.
lit int // The start position of any pending literal bytes.
)
for s+3 < len(src) {
// Update the hash table.
h := uint32(src[s]) | uint32(src[s+1])<<8 | uint32(src[s+2])<<16 | uint32(src[s+3])<<24
p := &e.table[(h*0x1e35a7bd)>>(32-tableBits)]
tmp := *p
p1 := int(tmp & 0xffffffff) // Closest match position
p2 := int(tmp >> 32) // Furthest match position
// We need to to store values in [-1, inf) in table.
// To save some initialization time, we make sure that
// e.cur is never zero.
t1 := p1 - e.cur
var l2 int
var t2 int
l1 := e.matchlen(s, t1, src)
// If fist match was ok, don't do the second.
if l1 < 16 {
t2 = p2 - e.cur
l2 = e.matchlen(s, t2, src)
// If both are short, continue
if l1 < 4 && l2 < 4 {
// Update hash table
*p = int64(s+e.cur) | (int64(p1) << 32)
// Skip 1 byte for 32 consecutive missed.
s += 1 + ((s - lit) >> 5)
continue
}
}
// Otherwise, we have a match. First, emit any pending literal bytes.
if lit != s {
emitLiteral(dst, src[lit:s])
}
// Update hash table
*p = int64(s+e.cur) | (int64(p1) << 32)
// Store the longest match l1 will be closest, so we prefer that if equal length
if l1 >= l2 {
dst.tokens[dst.n] = matchToken(uint32(l1-3), uint32(s-t1-minOffsetSize))
s += l1
} else {
dst.tokens[dst.n] = matchToken(uint32(l2-3), uint32(s-t2-minOffsetSize))
s += l2
}
dst.n++
lit = s
}
// Emit any final pending literal bytes and return.
if lit != len(src) {
emitLiteral(dst, src[lit:])
}
e.cur += len(src)
// Store this block, if it was full length.
if len(src) == maxStoreBlockSize {
copy(e.block[:], src)
e.prev = e.block[:len(src)]
} else {
e.prev = nil
}
}
func (e *snappyGen) matchlen(s, t int, src []byte) int {
// If t is invalid or src[s:s+4] differs from src[t:t+4], accumulate a literal byte.
offset := uint(s - t - 1)
// If we are inside the current block
if t >= 0 {
if offset >= (maxOffset-1) ||
src[s] != src[t] || src[s+1] != src[t+1] ||
src[s+2] != src[t+2] || src[s+3] != src[t+3] {
return 0
}
// Extend the match to be as long as possible.
s0 := s
s1 := s + maxMatchLength
if s1 > len(src) {
s1 = len(src)
}
s, t = s+4, t+4
for s < s1 && src[s] == src[t] {
s++
t++
}
return s - s0
}
// We found a match in the previous block.
tp := len(e.prev) + t
if tp < 0 || offset >= (maxOffset-1) || t > -5 ||
src[s] != e.prev[tp] || src[s+1] != e.prev[tp+1] ||
src[s+2] != e.prev[tp+2] || src[s+3] != e.prev[tp+3] {
return 0
}
// Extend the match to be as long as possible.
s0 := s
s1 := s + maxMatchLength
if s1 > len(src) {
s1 = len(src)
}
s, tp = s+4, tp+4
for s < s1 && src[s] == e.prev[tp] {
s++
tp++
if tp == len(e.prev) {
t = 0
// continue in current buffer
for s < s1 && src[s] == src[t] {
s++
t++
}
return s - s0
}
}
return s - s0
}
// Reset the encoding table.
func (e *snappyGen) Reset() {
e.prev = nil
}
// snappySSE4 extends snappyGen.
// This implementation can use SSE 4.2 for length matching.
type snappySSE4 struct {
snappyGen
}
// EncodeL3 uses a similar algorithm to level 2,
// but will keep two matches per hash.
// Both hashes are checked if the first isn't ok, and the longest is selected.
func (e *snappySSE4) encodeL3(dst *tokens, src []byte) {
// Return early if src is short.
if len(src) <= 4 {
if len(src) != 0 {
emitLiteral(dst, src)
}
e.prev = nil
e.cur += len(src)
return
}
// Ensure that e.cur doesn't wrap, mainly an issue on 32 bits.
if e.cur > 1<<30 {
e.cur = 1
}
// Iterate over the source bytes.
var (
s int // The iterator position.
lit int // The start position of any pending literal bytes.
)
for s+3 < len(src) {
// Load potential matches from hash table.
h := uint32(src[s]) | uint32(src[s+1])<<8 | uint32(src[s+2])<<16 | uint32(src[s+3])<<24
p := &e.table[(h*0x1e35a7bd)>>(32-tableBits)]
tmp := *p
p1 := int(tmp & 0xffffffff) // Closest match position
p2 := int(tmp >> 32) // Furthest match position
// We need to to store values in [-1, inf) in table.
// To save some initialization time, we make sure that
// e.cur is never zero.
t1 := int(p1) - e.cur
var l2 int
var t2 int
l1 := e.matchlen(s, t1, src)
// If fist match was ok, don't do the second.
if l1 < 16 {
t2 = int(p2) - e.cur
l2 = e.matchlen(s, t2, src)
// If both are short, continue
if l1 < 4 && l2 < 4 {
// Update hash table
*p = int64(s+e.cur) | (int64(p1) << 32)
// Skip 1 byte for 32 consecutive missed.
s += 1 + ((s - lit) >> 5)
continue
}
}
// Otherwise, we have a match. First, emit any pending literal bytes.
if lit != s {
emitLiteral(dst, src[lit:s])
}
// Update hash table
*p = int64(s+e.cur) | (int64(p1) << 32)
// Store the longest match l1 will be closest, so we prefer that if equal length
if l1 >= l2 {
dst.tokens[dst.n] = matchToken(uint32(l1-3), uint32(s-t1-minOffsetSize))
s += l1
} else {
dst.tokens[dst.n] = matchToken(uint32(l2-3), uint32(s-t2-minOffsetSize))
s += l2
}
dst.n++
lit = s
}
// Emit any final pending literal bytes and return.
if lit != len(src) {
emitLiteral(dst, src[lit:])
}
e.cur += len(src)
// Store this block, if it was full length.
if len(src) == maxStoreBlockSize {
copy(e.block[:], src)
e.prev = e.block[:len(src)]
} else {
e.prev = nil
}
}
func (e *snappySSE4) matchlen(s, t int, src []byte) int {
// If t is invalid or src[s:s+4] differs from src[t:t+4], accumulate a literal byte.
offset := uint(s - t - 1)
// If we are inside the current block
if t >= 0 {
if offset >= (maxOffset - 1) {
return 0
}
length := len(src) - s
if length > maxMatchLength {
length = maxMatchLength
}
// Extend the match to be as long as possible.
return matchLenSSE4(src[t:], src[s:], length)
}
// We found a match in the previous block.
tp := len(e.prev) + t
if tp < 0 || offset >= (maxOffset-1) || t > -5 ||
src[s] != e.prev[tp] || src[s+1] != e.prev[tp+1] ||
src[s+2] != e.prev[tp+2] || src[s+3] != e.prev[tp+3] {
return 0
}
// Extend the match to be as long as possible.
s0 := s
s1 := s + maxMatchLength
if s1 > len(src) {
s1 = len(src)
}
s, tp = s+4, tp+4
for s < s1 && src[s] == e.prev[tp] {
s++
tp++
if tp == len(e.prev) {
t = 0
// continue in current buffer
for s < s1 && src[s] == src[t] {
s++
t++
}
return s - s0
}
}
return s - s0
}

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-null-max.dyn.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-null-max.dyn.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-null-max.golden generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-null-max.in generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-null-max.wb.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-null-max.wb.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-pi.dyn.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-pi.dyn.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-pi.golden generated vendored Normal file
View File

Binary file not shown.

1
vendor/github.com/klauspost/compress/flate/testdata/huffman-pi.in generated vendored Normal file
View File

@@ -0,0 +1 @@
3.141592653589793238462643383279502884197169399375105820974944592307816406286208998628034825342117067982148086513282306647093844609550582231725359408128481117450284102701938521105559644622948954930381964428810975665933446128475648233786783165271201909145648566923460348610454326648213393607260249141273724587006606315588174881520920962829254091715364367892590360011330530548820466521384146951941511609433057270365759591953092186117381932611793105118548074462379962749567351885752724891227938183011949129833673362440656643086021394946395224737190702179860943702770539217176293176752384674818467669405132000568127145263560827785771342757789609173637178721468440901224953430146549585371050792279689258923542019956112129021960864034418159813629774771309960518707211349999998372978049951059731732816096318595024459455346908302642522308253344685035261931188171010003137838752886587533208381420617177669147303598253490428755468731159562863882353787593751957781857780532171226806613001927876611195909216420198938095257201065485863278865936153381827968230301952035301852968995773622599413891249721775283479131515574857242454150695950829533116861727855889075098381754637464939319255060400927701671139009848824012858361603563707660104710181942955596198946767837449448255379774726847104047534646208046684259069491293313677028989152104752162056966024058038150193511253382430035587640247496473263914199272604269922796782354781636009341721641219924586315030286182974555706749838505494588586926995690927210797509302955321165344987202755960236480665499119881834797753566369807426542527862551818417574672890977772793800081647060016145249192173217214772350141441973568548161361157352552133475741849468438523323907394143334547762416862518983569485562099219222184272550254256887671790494601653466804988627232791786085784383827967976681454100953883786360950680064225125205117392984896084128488626945604241965285022210661186306744278622039194945047123713786960956364371917287467764657573962413890865832645995813390478027590099465764078951269468398352595709825822620522489407726719478268482601476990902640136394437455305068203496252451749399651431429809190659250937221696461515709858387410597885959772975498930161753928468138268683868942774155991855925245953959431049972524680845987273644695848653836736222626099124608051243884390451244136549762780797715691435997700129616089441694868555848406353422072225828488648158456028506016842739452267467678895252138522549954666727823986456596116354886230577456498035593634568174324112515076069479451096596094025228879710893145669136867228748940560101503308617928680920874760917824938589009714909675985261365549781893129784821682998948722658804857564014270477555132379641451523746234364542858444795265867821051141354735739523113427166102135969536231442952484937187110145765403590279934403742007310578539062198387447808478489683321445713868751943506430218453191048481005370614680674919278191197939952061419663428754440643745123718192179998391015919561814675142691239748940907186494231961567945208095146550225231603881930142093762137855956638937787083039069792077346722182562599661501421503068038447734549202605414665925201497442850732518666002132434088190710486331734649651453905796268561005508106658796998163574736384052571459102897064140110971206280439039759515677157700420337869936007230558763176359421873125147120532928191826186125867321579198414848829164470609575270695722091756711672291098169091528017350671274858322287183520935396572512108357915136988209144421006751033467110314126711136990865851639831501970165151168517143765761835155650884909989859982387345528331635507647918535893226185489632132933089857064204675259070915481416549859461637180

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-pi.wb.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-pi.wb.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-1k.dyn.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-1k.dyn.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-1k.golden generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-1k.in generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-1k.wb.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-1k.wb.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-limit.dyn.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-limit.dyn.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-limit.golden generated vendored Normal file
View File

Binary file not shown.

4
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-limit.in generated vendored Normal file
View File

@@ -0,0 +1,4 @@
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
ř‹–vH
…”%€ŻÂţŤč ë†É·ĹŢę}‹ç>Úß˙lsŢĚçmŤIGH°čžň1YŢ4´[ĺŕ 0Â<30>[|]o#©
Ľ-#ľŮíul™ßýpfćîٱžn<C5BE>YŐÔ€Y<E282AC>w‰C8ÉŻ02š F=gn×ržN!OĆŕÔ{ŤĄö›kÜ*“w(ý´bÚ ç«kQC9/ ’lu>ô5ýC.÷¤uÚę›

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-limit.wb.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-limit.wb.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-max.golden generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-rand-max.in generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-shifts.dyn.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-shifts.dyn.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-shifts.golden generated vendored Normal file
View File

Binary file not shown.

2
vendor/github.com/klauspost/compress/flate/testdata/huffman-shifts.in generated vendored Normal file
View File

@@ -0,0 +1,2 @@
101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010
232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323232323

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-shifts.wb.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-shifts.wb.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-text-shift.dyn.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-text-shift.dyn.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-text-shift.golden generated vendored Normal file
View File

Binary file not shown.

14
vendor/github.com/klauspost/compress/flate/testdata/huffman-text-shift.in generated vendored Normal file
View File

@@ -0,0 +1,14 @@
//Copyright2009ThGoAuthor.Allrightrrvd.
//UofthiourccodigovrndbyBSD-tyl
//licnthtcnbfoundinthLICENSEfil.
pckgmin
import"o"
funcmin(){
vrb=mk([]byt,65535)
f,_:=o.Crt("huffmn-null-mx.in")
f.Writ(b)
}
ABCDEFGHIJKLMNOPQRSTUVXxyz!"#¤%&/?"

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-text-shift.wb.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-text-shift.wb.expect-noinput generated vendored Normal file
View File

Binary file not shown.

1
vendor/github.com/klauspost/compress/flate/testdata/huffman-text.dyn.expect generated vendored Normal file
View File

@@ -0,0 +1 @@
Ë_Kó0Åñëò½ê`KÇó0AasÄ›)^ˆHšþ²„¥IÉŸbß»¬—_>ç4

1
vendor/github.com/klauspost/compress/flate/testdata/huffman-text.dyn.expect-noinput generated vendored Normal file
View File

@@ -0,0 +1 @@
Ë_Kó0Åñëò½ê`KÇó0AasÄ›)^ˆHšþ²„¥IÉŸbß»¬—_>ç4

3
vendor/github.com/klauspost/compress/flate/testdata/huffman-text.golden generated vendored Normal file
View File

@@ -0,0 +1,3 @@
юAKС0ПСx╬ц÷·ZьзЯ╬LPьaн!┌x≥БADрЖI√&#I▀EЭНЧ гp]╒Lф©МЖ╞FПp≤╡ 1у88┤h⌠╒$┴ЁТ5SсЮ- ┌F66!┘)v┌.Т⌡0└Y╒≈М┘ШСц&Ее SсюыN|dё2:Ея
t≤|К▒█ЮЫИxz9÷═╜⌠ ┴И╙╨▀ё╡·┴и▌в3┼░
&&=Ыё╡╬╛Пц╢ UD▀=Fu▒РЦЁ]╡╛qЁшЩъUL+╫фНЖ╘>FQYйбLZ▐йoЭДэfTъ╣УEе╢рУ{╢Yй╤bЗeЗ

13
vendor/github.com/klauspost/compress/flate/testdata/huffman-text.in generated vendored Normal file
View File

@@ -0,0 +1,13 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package main
import "os"
func main() {
var b = make([]byte, 65535)
f, _ := os.Create("huffman-null-max.in")
f.Write(b)
}

1
vendor/github.com/klauspost/compress/flate/testdata/huffman-text.wb.expect generated vendored Normal file
View File

@@ -0,0 +1 @@
Ë_Kó0Åñëò½ê`KÇó0AasÄ›)^ˆHšþ²„¥IÉŸbß»¬—_>ç4

1
vendor/github.com/klauspost/compress/flate/testdata/huffman-text.wb.expect-noinput generated vendored Normal file
View File

@@ -0,0 +1 @@
Ë_Kó0Åñëò½ê`KÇó0AasÄ›)^ˆHšþ²„¥IÉŸbß»¬—_>ç4

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-zero.dyn.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-zero.dyn.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-zero.golden generated vendored Normal file
View File

Binary file not shown.

1
vendor/github.com/klauspost/compress/flate/testdata/huffman-zero.in generated vendored Normal file
View File

@@ -0,0 +1 @@
00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-zero.wb.expect generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/huffman-zero.wb.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/null-long-match.dyn.expect-noinput generated vendored Normal file
View File

Binary file not shown.

BIN
vendor/github.com/klauspost/compress/flate/testdata/null-long-match.wb.expect-noinput generated vendored Normal file
View File

Binary file not shown.

105
vendor/github.com/klauspost/compress/flate/token.go generated vendored Normal file
View File

@@ -0,0 +1,105 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
const (
// 2 bits: type 0 = literal 1=EOF 2=Match 3=Unused
// 8 bits: xlength = length - MIN_MATCH_LENGTH
// 22 bits xoffset = offset - MIN_OFFSET_SIZE, or literal
lengthShift = 22
offsetMask = 1<<lengthShift - 1
typeMask = 3 << 30
literalType = 0 << 30
matchType = 1 << 30
)
// The length code for length X (MIN_MATCH_LENGTH <= X <= MAX_MATCH_LENGTH)
// is lengthCodes[length - MIN_MATCH_LENGTH]
var lengthCodes = [...]uint32{
0, 1, 2, 3, 4, 5, 6, 7, 8, 8,
9, 9, 10, 10, 11, 11, 12, 12, 12, 12,
13, 13, 13, 13, 14, 14, 14, 14, 15, 15,
15, 15, 16, 16, 16, 16, 16, 16, 16, 16,
17, 17, 17, 17, 17, 17, 17, 17, 18, 18,
18, 18, 18, 18, 18, 18, 19, 19, 19, 19,
19, 19, 19, 19, 20, 20, 20, 20, 20, 20,
20, 20, 20, 20, 20, 20, 20, 20, 20, 20,
21, 21, 21, 21, 21, 21, 21, 21, 21, 21,
21, 21, 21, 21, 21, 21, 22, 22, 22, 22,
22, 22, 22, 22, 22, 22, 22, 22, 22, 22,
22, 22, 23, 23, 23, 23, 23, 23, 23, 23,
23, 23, 23, 23, 23, 23, 23, 23, 24, 24,
24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
24, 24, 24, 24, 24, 24, 24, 24, 24, 24,
25, 25, 25, 25, 25, 25, 25, 25, 25, 25,
25, 25, 25, 25, 25, 25, 25, 25, 25, 25,
25, 25, 25, 25, 25, 25, 25, 25, 25, 25,
25, 25, 26, 26, 26, 26, 26, 26, 26, 26,
26, 26, 26, 26, 26, 26, 26, 26, 26, 26,
26, 26, 26, 26, 26, 26, 26, 26, 26, 26,
26, 26, 26, 26, 27, 27, 27, 27, 27, 27,
27, 27, 27, 27, 27, 27, 27, 27, 27, 27,
27, 27, 27, 27, 27, 27, 27, 27, 27, 27,
27, 27, 27, 27, 27, 28,
}
var offsetCodes = [...]uint32{
0, 1, 2, 3, 4, 4, 5, 5, 6, 6, 6, 6, 7, 7, 7, 7,
8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9,
10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,
11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11, 11,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12,
13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13,
13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 13,
14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14,
14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14,
14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14,
14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14,
15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15,
15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15,
15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15,
15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15,
}
type token uint32
type tokens struct {
tokens []token
n int
}
// Convert a literal into a literal token.
func literalToken(literal uint32) token { return token(literalType + literal) }
// Convert a < xlength, xoffset > pair into a match token.
func matchToken(xlength uint32, xoffset uint32) token {
return token(matchType + xlength<<lengthShift + xoffset)
}
// Returns the type of a token
func (t token) typ() uint32 { return uint32(t) & typeMask }
// Returns the literal of a literal token
func (t token) literal() uint32 { return uint32(t - literalType) }
// Returns the extra offset of a match token
func (t token) offset() uint32 { return uint32(t) & offsetMask }
func (t token) length() uint32 { return uint32((t - matchType) >> lengthShift) }
func lengthCode(len uint32) uint32 { return lengthCodes[len] }
// Returns the offset code corresponding to a specific offset
func offsetCode(off uint32) uint32 {
if off < uint32(len(offsetCodes)) {
return offsetCodes[off]
} else if off>>7 < uint32(len(offsetCodes)) {
return offsetCodes[off>>7] + 14
} else {
return offsetCodes[off>>14] + 28
}
}

70
vendor/github.com/klauspost/compress/flate/writer_test.go generated vendored Normal file
View File

@@ -0,0 +1,70 @@
// Copyright 2012 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package flate
import (
"io/ioutil"
"runtime"
"testing"
)
func benchmarkEncoder(b *testing.B, testfile, level, n int) {
b.StopTimer()
b.SetBytes(int64(n))
buf0, err := ioutil.ReadFile(testfiles[testfile])
if err != nil {
b.Fatal(err)
}
if len(buf0) == 0 {
b.Fatalf("test file %q has no data", testfiles[testfile])
}
buf1 := make([]byte, n)
for i := 0; i < n; i += len(buf0) {
if len(buf0) > n-i {
buf0 = buf0[:n-i]
}
copy(buf1[i:], buf0)
}
buf0 = nil
runtime.GC()
w, err := NewWriter(ioutil.Discard, level)
b.StartTimer()
for i := 0; i < b.N; i++ {
w.Reset(ioutil.Discard)
_, err = w.Write(buf1)
if err != nil {
b.Fatal(err)
}
err = w.Close()
if err != nil {
b.Fatal(err)
}
}
}
func BenchmarkEncodeDigitsConstant1e4(b *testing.B) { benchmarkEncoder(b, digits, constant, 1e4) }
func BenchmarkEncodeDigitsConstant1e5(b *testing.B) { benchmarkEncoder(b, digits, constant, 1e5) }
func BenchmarkEncodeDigitsConstant1e6(b *testing.B) { benchmarkEncoder(b, digits, constant, 1e6) }
func BenchmarkEncodeDigitsSpeed1e4(b *testing.B) { benchmarkEncoder(b, digits, speed, 1e4) }
func BenchmarkEncodeDigitsSpeed1e5(b *testing.B) { benchmarkEncoder(b, digits, speed, 1e5) }
func BenchmarkEncodeDigitsSpeed1e6(b *testing.B) { benchmarkEncoder(b, digits, speed, 1e6) }
func BenchmarkEncodeDigitsDefault1e4(b *testing.B) { benchmarkEncoder(b, digits, default_, 1e4) }
func BenchmarkEncodeDigitsDefault1e5(b *testing.B) { benchmarkEncoder(b, digits, default_, 1e5) }
func BenchmarkEncodeDigitsDefault1e6(b *testing.B) { benchmarkEncoder(b, digits, default_, 1e6) }
func BenchmarkEncodeDigitsCompress1e4(b *testing.B) { benchmarkEncoder(b, digits, compress, 1e4) }
func BenchmarkEncodeDigitsCompress1e5(b *testing.B) { benchmarkEncoder(b, digits, compress, 1e5) }
func BenchmarkEncodeDigitsCompress1e6(b *testing.B) { benchmarkEncoder(b, digits, compress, 1e6) }
func BenchmarkEncodeTwainConstant1e4(b *testing.B) { benchmarkEncoder(b, twain, constant, 1e4) }
func BenchmarkEncodeTwainConstant1e5(b *testing.B) { benchmarkEncoder(b, twain, constant, 1e5) }
func BenchmarkEncodeTwainConstant1e6(b *testing.B) { benchmarkEncoder(b, twain, constant, 1e6) }
func BenchmarkEncodeTwainSpeed1e4(b *testing.B) { benchmarkEncoder(b, twain, speed, 1e4) }
func BenchmarkEncodeTwainSpeed1e5(b *testing.B) { benchmarkEncoder(b, twain, speed, 1e5) }
func BenchmarkEncodeTwainSpeed1e6(b *testing.B) { benchmarkEncoder(b, twain, speed, 1e6) }
func BenchmarkEncodeTwainDefault1e4(b *testing.B) { benchmarkEncoder(b, twain, default_, 1e4) }
func BenchmarkEncodeTwainDefault1e5(b *testing.B) { benchmarkEncoder(b, twain, default_, 1e5) }
func BenchmarkEncodeTwainDefault1e6(b *testing.B) { benchmarkEncoder(b, twain, default_, 1e6) }
func BenchmarkEncodeTwainCompress1e4(b *testing.B) { benchmarkEncoder(b, twain, compress, 1e4) }
func BenchmarkEncodeTwainCompress1e5(b *testing.B) { benchmarkEncoder(b, twain, compress, 1e5) }
func BenchmarkEncodeTwainCompress1e6(b *testing.B) { benchmarkEncoder(b, twain, compress, 1e6) }

342
vendor/github.com/klauspost/compress/gzip/gunzip.go generated vendored Normal file
View File

@@ -0,0 +1,342 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
// Package gzip implements reading and writing of gzip format compressed files,
// as specified in RFC 1952.
package gzip
import (
"bufio"
"errors"
"hash"
"io"
"time"
"github.com/klauspost/compress/flate"
"github.com/klauspost/crc32"
)
const (
gzipID1 = 0x1f
gzipID2 = 0x8b
gzipDeflate = 8
flagText = 1 << 0
flagHdrCrc = 1 << 1
flagExtra = 1 << 2
flagName = 1 << 3
flagComment = 1 << 4
)
func makeReader(r io.Reader) flate.Reader {
if rr, ok := r.(flate.Reader); ok {
return rr
}
return bufio.NewReader(r)
}
var (
// ErrChecksum is returned when reading GZIP data that has an invalid checksum.
ErrChecksum = errors.New("gzip: invalid checksum")
// ErrHeader is returned when reading GZIP data that has an invalid header.
ErrHeader = errors.New("gzip: invalid header")
)
// The gzip file stores a header giving metadata about the compressed file.
// That header is exposed as the fields of the Writer and Reader structs.
type Header struct {
Comment string // comment
Extra []byte // "extra data"
ModTime time.Time // modification time
Name string // file name
OS byte // operating system type
}
// A Reader is an io.Reader that can be read to retrieve
// uncompressed data from a gzip-format compressed file.
//
// In general, a gzip file can be a concatenation of gzip files,
// each with its own header. Reads from the Reader
// return the concatenation of the uncompressed data of each.
// Only the first header is recorded in the Reader fields.
//
// Gzip files store a length and checksum of the uncompressed data.
// The Reader will return a ErrChecksum when Read
// reaches the end of the uncompressed data if it does not
// have the expected length or checksum. Clients should treat data
// returned by Read as tentative until they receive the io.EOF
// marking the end of the data.
type Reader struct {
Header
r flate.Reader
decompressor io.ReadCloser
digest hash.Hash32
size uint32
flg byte
buf [512]byte
err error
multistream bool
}
// NewReader creates a new Reader reading the given reader.
// If r does not also implement io.ByteReader,
// the decompressor may read more data than necessary from r.
// It is the caller's responsibility to call Close on the Reader when done.
func NewReader(r io.Reader) (*Reader, error) {
z := new(Reader)
z.r = makeReader(r)
z.multistream = true
z.digest = crc32.NewIEEE()
if err := z.readHeader(true); err != nil {
return nil, err
}
return z, nil
}
// Reset discards the Reader z's state and makes it equivalent to the
// result of its original state from NewReader, but reading from r instead.
// This permits reusing a Reader rather than allocating a new one.
func (z *Reader) Reset(r io.Reader) error {
z.r = makeReader(r)
if z.digest == nil {
z.digest = crc32.NewIEEE()
} else {
z.digest.Reset()
}
z.size = 0
z.err = nil
z.multistream = true
return z.readHeader(true)
}
// Multistream controls whether the reader supports multistream files.
//
// If enabled (the default), the Reader expects the input to be a sequence
// of individually gzipped data streams, each with its own header and
// trailer, ending at EOF. The effect is that the concatenation of a sequence
// of gzipped files is treated as equivalent to the gzip of the concatenation
// of the sequence. This is standard behavior for gzip readers.
//
// Calling Multistream(false) disables this behavior; disabling the behavior
// can be useful when reading file formats that distinguish individual gzip
// data streams or mix gzip data streams with other data streams.
// In this mode, when the Reader reaches the end of the data stream,
// Read returns io.EOF. If the underlying reader implements io.ByteReader,
// it will be left positioned just after the gzip stream.
// To start the next stream, call z.Reset(r) followed by z.Multistream(false).
// If there is no next stream, z.Reset(r) will return io.EOF.
func (z *Reader) Multistream(ok bool) {
z.multistream = ok
}
// GZIP (RFC 1952) is little-endian, unlike ZLIB (RFC 1950).
func get4(p []byte) uint32 {
return uint32(p[0]) | uint32(p[1])<<8 | uint32(p[2])<<16 | uint32(p[3])<<24
}
func (z *Reader) readString() (string, error) {
var err error
needconv := false
for i := 0; ; i++ {
if i >= len(z.buf) {
return "", ErrHeader
}
z.buf[i], err = z.r.ReadByte()
if err != nil {
return "", err
}
if z.buf[i] > 0x7f {
needconv = true
}
if z.buf[i] == 0 {
// GZIP (RFC 1952) specifies that strings are NUL-terminated ISO 8859-1 (Latin-1).
if needconv {
s := make([]rune, 0, i)
for _, v := range z.buf[0:i] {
s = append(s, rune(v))
}
return string(s), nil
}
return string(z.buf[0:i]), nil
}
}
}
func (z *Reader) read2() (uint32, error) {
_, err := io.ReadFull(z.r, z.buf[0:2])
if err != nil {
return 0, err
}
return uint32(z.buf[0]) | uint32(z.buf[1])<<8, nil
}
func (z *Reader) readHeader(save bool) error {
_, err := io.ReadFull(z.r, z.buf[0:10])
if err != nil {
return err
}
if z.buf[0] != gzipID1 || z.buf[1] != gzipID2 || z.buf[2] != gzipDeflate {
return ErrHeader
}
z.flg = z.buf[3]
if save {
z.ModTime = time.Unix(int64(get4(z.buf[4:8])), 0)
// z.buf[8] is xfl, ignored
z.OS = z.buf[9]
}
z.digest.Reset()
z.digest.Write(z.buf[0:10])
if z.flg&flagExtra != 0 {
n, err := z.read2()
if err != nil {
return err
}
data := make([]byte, n)
if _, err = io.ReadFull(z.r, data); err != nil {
return err
}
if save {
z.Extra = data
}
}
var s string
if z.flg&flagName != 0 {
if s, err = z.readString(); err != nil {
return err
}
if save {
z.Name = s
}
}
if z.flg&flagComment != 0 {
if s, err = z.readString(); err != nil {
return err
}
if save {
z.Comment = s
}
}
if z.flg&flagHdrCrc != 0 {
n, err := z.read2()
if err != nil {
return err
}
sum := z.digest.Sum32() & 0xFFFF
if n != sum {
return ErrHeader
}
}
z.digest.Reset()
if z.decompressor == nil {
z.decompressor = flate.NewReader(z.r)
} else {
z.decompressor.(flate.Resetter).Reset(z.r, nil)
}
return nil
}
func (z *Reader) Read(p []byte) (n int, err error) {
if z.err != nil {
return 0, z.err
}
if len(p) == 0 {
return 0, nil
}
n, err = z.decompressor.Read(p)
z.digest.Write(p[0:n])
z.size += uint32(n)
if n != 0 || err != io.EOF {
z.err = err
return
}
// Finished file; check checksum + size.
if _, err := io.ReadFull(z.r, z.buf[0:8]); err != nil {
z.err = err
return 0, err
}
crc32, isize := get4(z.buf[0:4]), get4(z.buf[4:8])
sum := z.digest.Sum32()
if sum != crc32 || isize != z.size {
z.err = ErrChecksum
return 0, z.err
}
// File is ok; is there another?
if !z.multistream {
return 0, io.EOF
}
if err = z.readHeader(false); err != nil {
z.err = err
return
}
// Yes. Reset and read from it.
z.digest.Reset()
z.size = 0
return z.Read(p)
}
// Support the io.WriteTo interface for io.Copy and friends.
func (z *Reader) WriteTo(w io.Writer) (int64, error) {
total := int64(0)
for {
if z.err != nil {
if z.err == io.EOF {
return total, nil
}
return total, z.err
}
// We write both to output and digest.
mw := io.MultiWriter(w, z.digest)
n, err := z.decompressor.(io.WriterTo).WriteTo(mw)
total += n
z.size += uint32(n)
if err != nil {
z.err = err
return total, z.err
}
// Finished file; check checksum + size.
if _, err := io.ReadFull(z.r, z.buf[0:8]); err != nil {
z.err = err
return 0, err
}
crc32, isize := get4(z.buf[0:4]), get4(z.buf[4:8])
sum := z.digest.Sum32()
if sum != crc32 || isize != z.size {
z.err = ErrChecksum
return 0, z.err
}
// File is ok; is there another?
if !z.multistream {
return total, nil
}
err = z.readHeader(false)
// There was not more
if err == io.EOF {
return total, nil
}
if err != nil {
z.err = err
return total, err
}
// Yes. Reset and read from it.
z.digest.Reset()
z.size = 0
}
}
// Close closes the Reader. It does not close the underlying io.Reader.
func (z *Reader) Close() error { return z.decompressor.Close() }

571
vendor/github.com/klauspost/compress/gzip/gunzip_test.go generated vendored Normal file
View File

@@ -0,0 +1,571 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package gzip
import (
"bytes"
oldgz "compress/gzip"
"crypto/rand"
"io"
"io/ioutil"
"os"
"strings"
"testing"
"time"
)
type gunzipTest struct {
name string
desc string
raw string
gzip []byte
err error
}
var gunzipTests = []gunzipTest{
{ // has 1 empty fixed-huffman block
"empty.txt",
"empty.txt",
"",
[]byte{
0x1f, 0x8b, 0x08, 0x08, 0xf7, 0x5e, 0x14, 0x4a,
0x00, 0x03, 0x65, 0x6d, 0x70, 0x74, 0x79, 0x2e,
0x74, 0x78, 0x74, 0x00, 0x03, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
},
nil,
},
{ // has 1 non-empty fixed huffman block
"hello.txt",
"hello.txt",
"hello world\n",
[]byte{
0x1f, 0x8b, 0x08, 0x08, 0xc8, 0x58, 0x13, 0x4a,
0x00, 0x03, 0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x2e,
0x74, 0x78, 0x74, 0x00, 0xcb, 0x48, 0xcd, 0xc9,
0xc9, 0x57, 0x28, 0xcf, 0x2f, 0xca, 0x49, 0xe1,
0x02, 0x00, 0x2d, 0x3b, 0x08, 0xaf, 0x0c, 0x00,
0x00, 0x00,
},
nil,
},
{ // concatenation
"hello.txt",
"hello.txt x2",
"hello world\n" +
"hello world\n",
[]byte{
0x1f, 0x8b, 0x08, 0x08, 0xc8, 0x58, 0x13, 0x4a,
0x00, 0x03, 0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x2e,
0x74, 0x78, 0x74, 0x00, 0xcb, 0x48, 0xcd, 0xc9,
0xc9, 0x57, 0x28, 0xcf, 0x2f, 0xca, 0x49, 0xe1,
0x02, 0x00, 0x2d, 0x3b, 0x08, 0xaf, 0x0c, 0x00,
0x00, 0x00,
0x1f, 0x8b, 0x08, 0x08, 0xc8, 0x58, 0x13, 0x4a,
0x00, 0x03, 0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x2e,
0x74, 0x78, 0x74, 0x00, 0xcb, 0x48, 0xcd, 0xc9,
0xc9, 0x57, 0x28, 0xcf, 0x2f, 0xca, 0x49, 0xe1,
0x02, 0x00, 0x2d, 0x3b, 0x08, 0xaf, 0x0c, 0x00,
0x00, 0x00,
},
nil,
},
{ // has a fixed huffman block with some length-distance pairs
"shesells.txt",
"shesells.txt",
"she sells seashells by the seashore\n",
[]byte{
0x1f, 0x8b, 0x08, 0x08, 0x72, 0x66, 0x8b, 0x4a,
0x00, 0x03, 0x73, 0x68, 0x65, 0x73, 0x65, 0x6c,
0x6c, 0x73, 0x2e, 0x74, 0x78, 0x74, 0x00, 0x2b,
0xce, 0x48, 0x55, 0x28, 0x4e, 0xcd, 0xc9, 0x29,
0x06, 0x92, 0x89, 0xc5, 0x19, 0x60, 0x56, 0x52,
0xa5, 0x42, 0x09, 0x58, 0x18, 0x28, 0x90, 0x5f,
0x94, 0xca, 0x05, 0x00, 0x76, 0xb0, 0x3b, 0xeb,
0x24, 0x00, 0x00, 0x00,
},
nil,
},
{ // has dynamic huffman blocks
"gettysburg",
"gettysburg",
" Four score and seven years ago our fathers brought forth on\n" +
"this continent, a new nation, conceived in Liberty, and dedicated\n" +
"to the proposition that all men are created equal.\n" +
" Now we are engaged in a great Civil War, testing whether that\n" +
"nation, or any nation so conceived and so dedicated, can long\n" +
"endure.\n" +
" We are met on a great battle-field of that war.\n" +
" We have come to dedicate a portion of that field, as a final\n" +
"resting place for those who here gave their lives that that\n" +
"nation might live. It is altogether fitting and proper that\n" +
"we should do this.\n" +
" But, in a larger sense, we can not dedicate — we can not\n" +
"consecrate — we can not hallow — this ground.\n" +
" The brave men, living and dead, who struggled here, have\n" +
"consecrated it, far above our poor power to add or detract.\n" +
"The world will little note, nor long remember what we say here,\n" +
"but it can never forget what they did here.\n" +
" It is for us the living, rather, to be dedicated here to the\n" +
"unfinished work which they who fought here have thus far so\n" +
"nobly advanced. It is rather for us to be here dedicated to\n" +
"the great task remaining before us — that from these honored\n" +
"dead we take increased devotion to that cause for which they\n" +
"gave the last full measure of devotion —\n" +
" that we here highly resolve that these dead shall not have\n" +
"died in vain — that this nation, under God, shall have a new\n" +
"birth of freedom — and that government of the people, by the\n" +
"people, for the people, shall not perish from this earth.\n" +
"\n" +
"Abraham Lincoln, November 19, 1863, Gettysburg, Pennsylvania\n",
[]byte{
0x1f, 0x8b, 0x08, 0x08, 0xd1, 0x12, 0x2b, 0x4a,
0x00, 0x03, 0x67, 0x65, 0x74, 0x74, 0x79, 0x73,
0x62, 0x75, 0x72, 0x67, 0x00, 0x65, 0x54, 0xcd,
0x6e, 0xd4, 0x30, 0x10, 0xbe, 0xfb, 0x29, 0xe6,
0x01, 0x42, 0xa5, 0x0a, 0x09, 0xc1, 0x11, 0x90,
0x40, 0x48, 0xa8, 0xe2, 0x80, 0xd4, 0xf3, 0x24,
0x9e, 0x24, 0x56, 0xbd, 0x9e, 0xc5, 0x76, 0x76,
0x95, 0x1b, 0x0f, 0xc1, 0x13, 0xf2, 0x24, 0x7c,
0x63, 0x77, 0x9b, 0x4a, 0x5c, 0xaa, 0x6e, 0x6c,
0xcf, 0x7c, 0x7f, 0x33, 0x44, 0x5f, 0x74, 0xcb,
0x54, 0x26, 0xcd, 0x42, 0x9c, 0x3c, 0x15, 0xb9,
0x48, 0xa2, 0x5d, 0x38, 0x17, 0xe2, 0x45, 0xc9,
0x4e, 0x67, 0xae, 0xab, 0xe0, 0xf7, 0x98, 0x75,
0x5b, 0xd6, 0x4a, 0xb3, 0xe6, 0xba, 0x92, 0x26,
0x57, 0xd7, 0x50, 0x68, 0xd2, 0x54, 0x43, 0x92,
0x54, 0x07, 0x62, 0x4a, 0x72, 0xa5, 0xc4, 0x35,
0x68, 0x1a, 0xec, 0x60, 0x92, 0x70, 0x11, 0x4f,
0x21, 0xd1, 0xf7, 0x30, 0x4a, 0xae, 0xfb, 0xd0,
0x9a, 0x78, 0xf1, 0x61, 0xe2, 0x2a, 0xde, 0x55,
0x25, 0xd4, 0xa6, 0x73, 0xd6, 0xb3, 0x96, 0x60,
0xef, 0xf0, 0x9b, 0x2b, 0x71, 0x8c, 0x74, 0x02,
0x10, 0x06, 0xac, 0x29, 0x8b, 0xdd, 0x25, 0xf9,
0xb5, 0x71, 0xbc, 0x73, 0x44, 0x0f, 0x7a, 0xa5,
0xab, 0xb4, 0x33, 0x49, 0x0b, 0x2f, 0xbd, 0x03,
0xd3, 0x62, 0x17, 0xe9, 0x73, 0xb8, 0x84, 0x48,
0x8f, 0x9c, 0x07, 0xaa, 0x52, 0x00, 0x6d, 0xa1,
0xeb, 0x2a, 0xc6, 0xa0, 0x95, 0x76, 0x37, 0x78,
0x9a, 0x81, 0x65, 0x7f, 0x46, 0x4b, 0x45, 0x5f,
0xe1, 0x6d, 0x42, 0xe8, 0x01, 0x13, 0x5c, 0x38,
0x51, 0xd4, 0xb4, 0x38, 0x49, 0x7e, 0xcb, 0x62,
0x28, 0x1e, 0x3b, 0x82, 0x93, 0x54, 0x48, 0xf1,
0xd2, 0x7d, 0xe4, 0x5a, 0xa3, 0xbc, 0x99, 0x83,
0x44, 0x4f, 0x3a, 0x77, 0x36, 0x57, 0xce, 0xcf,
0x2f, 0x56, 0xbe, 0x80, 0x90, 0x9e, 0x84, 0xea,
0x51, 0x1f, 0x8f, 0xcf, 0x90, 0xd4, 0x60, 0xdc,
0x5e, 0xb4, 0xf7, 0x10, 0x0b, 0x26, 0xe0, 0xff,
0xc4, 0xd1, 0xe5, 0x67, 0x2e, 0xe7, 0xc8, 0x93,
0x98, 0x05, 0xb8, 0xa8, 0x45, 0xc0, 0x4d, 0x09,
0xdc, 0x84, 0x16, 0x2b, 0x0d, 0x9a, 0x21, 0x53,
0x04, 0x8b, 0xd2, 0x0b, 0xbd, 0xa2, 0x4c, 0xa7,
0x60, 0xee, 0xd9, 0xe1, 0x1d, 0xd1, 0xb7, 0x4a,
0x30, 0x8f, 0x63, 0xd5, 0xa5, 0x8b, 0x33, 0x87,
0xda, 0x1a, 0x18, 0x79, 0xf3, 0xe3, 0xa6, 0x17,
0x94, 0x2e, 0xab, 0x6e, 0xa0, 0xe3, 0xcd, 0xac,
0x50, 0x8c, 0xca, 0xa7, 0x0d, 0x76, 0x37, 0xd1,
0x23, 0xe7, 0x05, 0x57, 0x8b, 0xa4, 0x22, 0x83,
0xd9, 0x62, 0x52, 0x25, 0xad, 0x07, 0xbb, 0xbf,
0xbf, 0xff, 0xbc, 0xfa, 0xee, 0x20, 0x73, 0x91,
0x29, 0xff, 0x7f, 0x02, 0x71, 0x62, 0x84, 0xb5,
0xf6, 0xb5, 0x25, 0x6b, 0x41, 0xde, 0x92, 0xb7,
0x76, 0x3f, 0x91, 0x91, 0x31, 0x1b, 0x41, 0x84,
0x62, 0x30, 0x0a, 0x37, 0xa4, 0x5e, 0x18, 0x3a,
0x99, 0x08, 0xa5, 0xe6, 0x6d, 0x59, 0x22, 0xec,
0x33, 0x39, 0x86, 0x26, 0xf5, 0xab, 0x66, 0xc8,
0x08, 0x20, 0xcf, 0x0c, 0xd7, 0x47, 0x45, 0x21,
0x0b, 0xf6, 0x59, 0xd5, 0xfe, 0x5c, 0x8d, 0xaa,
0x12, 0x7b, 0x6f, 0xa1, 0xf0, 0x52, 0x33, 0x4f,
0xf5, 0xce, 0x59, 0xd3, 0xab, 0x66, 0x10, 0xbf,
0x06, 0xc4, 0x31, 0x06, 0x73, 0xd6, 0x80, 0xa2,
0x78, 0xc2, 0x45, 0xcb, 0x03, 0x65, 0x39, 0xc9,
0x09, 0xd1, 0x06, 0x04, 0x33, 0x1a, 0x5a, 0xf1,
0xde, 0x01, 0xb8, 0x71, 0x83, 0xc4, 0xb5, 0xb3,
0xc3, 0x54, 0x65, 0x33, 0x0d, 0x5a, 0xf7, 0x9b,
0x90, 0x7c, 0x27, 0x1f, 0x3a, 0x58, 0xa3, 0xd8,
0xfd, 0x30, 0x5f, 0xb7, 0xd2, 0x66, 0xa2, 0x93,
0x1c, 0x28, 0xb7, 0xe9, 0x1b, 0x0c, 0xe1, 0x28,
0x47, 0x26, 0xbb, 0xe9, 0x7d, 0x7e, 0xdc, 0x96,
0x10, 0x92, 0x50, 0x56, 0x7c, 0x06, 0xe2, 0x27,
0xb4, 0x08, 0xd3, 0xda, 0x7b, 0x98, 0x34, 0x73,
0x9f, 0xdb, 0xf6, 0x62, 0xed, 0x31, 0x41, 0x13,
0xd3, 0xa2, 0xa8, 0x4b, 0x3a, 0xc6, 0x1d, 0xe4,
0x2f, 0x8c, 0xf8, 0xfb, 0x97, 0x64, 0xf4, 0xb6,
0x2f, 0x80, 0x5a, 0xf3, 0x56, 0xe0, 0x40, 0x50,
0xd5, 0x19, 0xd0, 0x1e, 0xfc, 0xca, 0xe5, 0xc9,
0xd4, 0x60, 0x00, 0x81, 0x2e, 0xa3, 0xcc, 0xb6,
0x52, 0xf0, 0xb4, 0xdb, 0x69, 0x99, 0xce, 0x7a,
0x32, 0x4c, 0x08, 0xed, 0xaa, 0x10, 0x10, 0xe3,
0x6f, 0xee, 0x99, 0x68, 0x95, 0x9f, 0x04, 0x71,
0xb2, 0x49, 0x2f, 0x62, 0xa6, 0x5e, 0xb4, 0xef,
0x02, 0xed, 0x4f, 0x27, 0xde, 0x4a, 0x0f, 0xfd,
0xc1, 0xcc, 0xdd, 0x02, 0x8f, 0x08, 0x16, 0x54,
0xdf, 0xda, 0xca, 0xe0, 0x82, 0xf1, 0xb4, 0x31,
0x7a, 0xa9, 0x81, 0xfe, 0x90, 0xb7, 0x3e, 0xdb,
0xd3, 0x35, 0xc0, 0x20, 0x80, 0x33, 0x46, 0x4a,
0x63, 0xab, 0xd1, 0x0d, 0x29, 0xd2, 0xe2, 0x84,
0xb8, 0xdb, 0xfa, 0xe9, 0x89, 0x44, 0x86, 0x7c,
0xe8, 0x0b, 0xe6, 0x02, 0x6a, 0x07, 0x9b, 0x96,
0xd0, 0xdb, 0x2e, 0x41, 0x4c, 0xa1, 0xd5, 0x57,
0x45, 0x14, 0xfb, 0xe3, 0xa6, 0x72, 0x5b, 0x87,
0x6e, 0x0c, 0x6d, 0x5b, 0xce, 0xe0, 0x2f, 0xe2,
0x21, 0x81, 0x95, 0xb0, 0xe8, 0xb6, 0x32, 0x0b,
0xb2, 0x98, 0x13, 0x52, 0x5d, 0xfb, 0xec, 0x63,
0x17, 0x8a, 0x9e, 0x23, 0x22, 0x36, 0xee, 0xcd,
0xda, 0xdb, 0xcf, 0x3e, 0xf1, 0xc7, 0xf1, 0x01,
0x12, 0x93, 0x0a, 0xeb, 0x6f, 0xf2, 0x02, 0x15,
0x96, 0x77, 0x5d, 0xef, 0x9c, 0xfb, 0x88, 0x91,
0x59, 0xf9, 0x84, 0xdd, 0x9b, 0x26, 0x8d, 0x80,
0xf9, 0x80, 0x66, 0x2d, 0xac, 0xf7, 0x1f, 0x06,
0xba, 0x7f, 0xff, 0xee, 0xed, 0x40, 0x5f, 0xa5,
0xd6, 0xbd, 0x8c, 0x5b, 0x46, 0xd2, 0x7e, 0x48,
0x4a, 0x65, 0x8f, 0x08, 0x42, 0x60, 0xf7, 0x0f,
0xb9, 0x16, 0x0b, 0x0c, 0x1a, 0x06, 0x00, 0x00,
},
nil,
},
{ // has 1 non-empty fixed huffman block then garbage
"hello.txt",
"hello.txt + garbage",
"hello world\n",
[]byte{
0x1f, 0x8b, 0x08, 0x08, 0xc8, 0x58, 0x13, 0x4a,
0x00, 0x03, 0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x2e,
0x74, 0x78, 0x74, 0x00, 0xcb, 0x48, 0xcd, 0xc9,
0xc9, 0x57, 0x28, 0xcf, 0x2f, 0xca, 0x49, 0xe1,
0x02, 0x00, 0x2d, 0x3b, 0x08, 0xaf, 0x0c, 0x00,
0x00, 0x00, 'g', 'a', 'r', 'b', 'a', 'g', 'e', '!', '!', '!',
},
ErrHeader,
},
{ // has 1 non-empty fixed huffman block not enough header
"hello.txt",
"hello.txt + garbage",
"hello world\n",
[]byte{
0x1f, 0x8b, 0x08, 0x08, 0xc8, 0x58, 0x13, 0x4a,
0x00, 0x03, 0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x2e,
0x74, 0x78, 0x74, 0x00, 0xcb, 0x48, 0xcd, 0xc9,
0xc9, 0x57, 0x28, 0xcf, 0x2f, 0xca, 0x49, 0xe1,
0x02, 0x00, 0x2d, 0x3b, 0x08, 0xaf, 0x0c, 0x00,
0x00, 0x00, gzipID1,
},
io.ErrUnexpectedEOF,
},
{ // has 1 non-empty fixed huffman block but corrupt checksum
"hello.txt",
"hello.txt + corrupt checksum",
"hello world\n",
[]byte{
0x1f, 0x8b, 0x08, 0x08, 0xc8, 0x58, 0x13, 0x4a,
0x00, 0x03, 0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x2e,
0x74, 0x78, 0x74, 0x00, 0xcb, 0x48, 0xcd, 0xc9,
0xc9, 0x57, 0x28, 0xcf, 0x2f, 0xca, 0x49, 0xe1,
0x02, 0x00, 0xff, 0xff, 0xff, 0xff, 0x0c, 0x00,
0x00, 0x00,
},
ErrChecksum,
},
{ // has 1 non-empty fixed huffman block but corrupt size
"hello.txt",
"hello.txt + corrupt size",
"hello world\n",
[]byte{
0x1f, 0x8b, 0x08, 0x08, 0xc8, 0x58, 0x13, 0x4a,
0x00, 0x03, 0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x2e,
0x74, 0x78, 0x74, 0x00, 0xcb, 0x48, 0xcd, 0xc9,
0xc9, 0x57, 0x28, 0xcf, 0x2f, 0xca, 0x49, 0xe1,
0x02, 0x00, 0x2d, 0x3b, 0x08, 0xaf, 0xff, 0x00,
0x00, 0x00,
},
ErrChecksum,
},
}
func TestDecompressor(t *testing.T) {
b := new(bytes.Buffer)
for _, tt := range gunzipTests {
in := bytes.NewReader(tt.gzip)
gzip, err := NewReader(in)
if err != nil {
t.Errorf("%s: NewReader: %s", tt.name, err)
continue
}
defer gzip.Close()
if tt.name != gzip.Name {
t.Errorf("%s: got name %s", tt.name, gzip.Name)
}
b.Reset()
n, err := io.Copy(b, gzip)
if err != tt.err {
t.Errorf("%s: io.Copy: %v want %v", tt.name, err, tt.err)
}
s := b.String()
if s != tt.raw {
t.Errorf("%s: got %d-byte %q want %d-byte %q", tt.name, n, s, len(tt.raw), tt.raw)
}
// Test Reader Reset.
in = bytes.NewReader(tt.gzip)
err = gzip.Reset(in)
if err != nil {
t.Errorf("%s: Reset: %s", tt.name, err)
continue
}
if tt.name != gzip.Name {
t.Errorf("%s: got name %s", tt.name, gzip.Name)
}
b.Reset()
n, err = io.Copy(b, gzip)
if err != tt.err {
t.Errorf("%s: io.Copy: %v want %v", tt.name, err, tt.err)
}
s = b.String()
if s != tt.raw {
t.Errorf("%s: got %d-byte %q want %d-byte %q", tt.name, n, s, len(tt.raw), tt.raw)
}
}
}
func TestIssue6550(t *testing.T) {
f, err := os.Open("testdata/issue6550.gz")
if err != nil {
t.Fatal(err)
}
gzip, err := NewReader(f)
if err != nil {
t.Fatalf("NewReader(testdata/issue6550.gz): %v", err)
}
defer gzip.Close()
done := make(chan bool, 1)
go func() {
_, err := io.Copy(ioutil.Discard, gzip)
if err == nil {
t.Errorf("Copy succeeded")
} else {
t.Logf("Copy failed (correctly): %v", err)
}
done <- true
}()
select {
case <-time.After(1 * time.Second):
t.Errorf("Copy hung")
case <-done:
// ok
}
}
func TestInitialReset(t *testing.T) {
var r Reader
if err := r.Reset(bytes.NewReader(gunzipTests[1].gzip)); err != nil {
t.Error(err)
}
var buf bytes.Buffer
if _, err := io.Copy(&buf, &r); err != nil {
t.Error(err)
}
if s := buf.String(); s != gunzipTests[1].raw {
t.Errorf("got %q want %q", s, gunzipTests[1].raw)
}
}
func TestMultistreamFalse(t *testing.T) {
// Find concatenation test.
var tt gunzipTest
for _, tt = range gunzipTests {
if strings.HasSuffix(tt.desc, " x2") {
goto Found
}
}
t.Fatal("cannot find hello.txt x2 in gunzip tests")
Found:
br := bytes.NewReader(tt.gzip)
var r Reader
if err := r.Reset(br); err != nil {
t.Fatalf("first reset: %v", err)
}
// Expect two streams with "hello world\n", then real EOF.
const hello = "hello world\n"
r.Multistream(false)
data, err := ioutil.ReadAll(&r)
if string(data) != hello || err != nil {
t.Fatalf("first stream = %q, %v, want %q, %v", string(data), err, hello, nil)
}
if err := r.Reset(br); err != nil {
t.Fatalf("second reset: %v", err)
}
r.Multistream(false)
data, err = ioutil.ReadAll(&r)
if string(data) != hello || err != nil {
t.Fatalf("second stream = %q, %v, want %q, %v", string(data), err, hello, nil)
}
if err := r.Reset(br); err != io.EOF {
t.Fatalf("third reset: err=%v, want io.EOF", err)
}
}
func TestWriteTo(t *testing.T) {
input := make([]byte, 100000)
n, err := rand.Read(input)
if err != nil {
t.Fatal(err)
}
if n != len(input) {
t.Fatal("did not fill buffer")
}
compressed := &bytes.Buffer{}
// Do it twice to test MultiStream functionality
for i := 0; i < 2; i++ {
w, err := NewWriterLevel(compressed, -2)
if err != nil {
t.Fatal(err)
}
n, err = w.Write(input)
if err != nil {
t.Fatal(err)
}
if n != len(input) {
t.Fatal("did not fill buffer")
}
w.Close()
}
input = append(input, input...)
buf := compressed.Bytes()
dec, err := NewReader(bytes.NewBuffer(buf))
if err != nil {
t.Fatal(err)
}
// ReadAll does not use WriteTo, but we wrap it in a NopCloser to be sure.
readall, err := ioutil.ReadAll(ioutil.NopCloser(dec))
if err != nil {
t.Fatal(err)
}
if len(readall) != len(input) {
t.Fatal("did not decompress everything")
}
if bytes.Compare(readall, input) != 0 {
t.Fatal("output did not match input")
}
dec, err = NewReader(bytes.NewBuffer(buf))
if err != nil {
t.Fatal(err)
}
wtbuf := &bytes.Buffer{}
written, err := dec.WriteTo(wtbuf)
if err != nil {
t.Fatal(err)
}
if written != int64(len(input)) {
t.Error("Returned length did not match, expected", len(input), "got", written)
}
if wtbuf.Len() != len(input) {
t.Error("Actual Length did not match, expected", len(input), "got", wtbuf.Len())
}
if bytes.Compare(wtbuf.Bytes(), input) != 0 {
t.Fatal("output did not match input")
}
}
func BenchmarkGunzipCopy(b *testing.B) {
dat, _ := ioutil.ReadFile("testdata/test.json")
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
dst := &bytes.Buffer{}
w, _ := NewWriterLevel(dst, 1)
_, err := w.Write(dat)
if err != nil {
b.Fatal(err)
}
w.Close()
input := dst.Bytes()
b.SetBytes(int64(len(dat)))
b.ResetTimer()
for n := 0; n < b.N; n++ {
r, err := NewReader(bytes.NewBuffer(input))
if err != nil {
b.Fatal(err)
}
_, err = io.Copy(ioutil.Discard, r)
if err != nil {
b.Fatal(err)
}
}
}
func BenchmarkGunzipNoWriteTo(b *testing.B) {
dat, _ := ioutil.ReadFile("testdata/test.json")
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
dst := &bytes.Buffer{}
w, _ := NewWriterLevel(dst, 1)
_, err := w.Write(dat)
if err != nil {
b.Fatal(err)
}
w.Close()
input := dst.Bytes()
r, err := NewReader(bytes.NewBuffer(input))
if err != nil {
b.Fatal(err)
}
b.SetBytes(int64(len(dat)))
b.ResetTimer()
for n := 0; n < b.N; n++ {
err := r.Reset(bytes.NewBuffer(input))
if err != nil {
b.Fatal(err)
}
_, err = io.Copy(ioutil.Discard, ioutil.NopCloser(r))
if err != nil {
b.Fatal(err)
}
}
}
func BenchmarkGunzipStdlib(b *testing.B) {
dat, _ := ioutil.ReadFile("testdata/test.json")
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
dst := &bytes.Buffer{}
w, _ := NewWriterLevel(dst, 1)
_, err := w.Write(dat)
if err != nil {
b.Fatal(err)
}
w.Close()
input := dst.Bytes()
r, err := oldgz.NewReader(bytes.NewBuffer(input))
if err != nil {
b.Fatal(err)
}
b.SetBytes(int64(len(dat)))
b.ResetTimer()
for n := 0; n < b.N; n++ {
err := r.Reset(bytes.NewBuffer(input))
if err != nil {
b.Fatal(err)
}
_, err = io.Copy(ioutil.Discard, r)
if err != nil {
b.Fatal(err)
}
}
}

274
vendor/github.com/klauspost/compress/gzip/gzip.go generated vendored Normal file
View File

@@ -0,0 +1,274 @@
// Copyright 2010 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package gzip
import (
"errors"
"fmt"
"hash"
"io"
"github.com/klauspost/compress/flate"
"github.com/klauspost/crc32"
)
// These constants are copied from the flate package, so that code that imports
// "compress/gzip" does not also have to import "compress/flate".
const (
NoCompression = flate.NoCompression
BestSpeed = flate.BestSpeed
BestCompression = flate.BestCompression
DefaultCompression = flate.DefaultCompression
ConstantCompression = flate.ConstantCompression
)
// A Writer is an io.WriteCloser.
// Writes to a Writer are compressed and written to w.
type Writer struct {
Header
w io.Writer
level int
wroteHeader bool
compressor *flate.Writer
digest hash.Hash32
size uint32
closed bool
buf [10]byte
err error
}
// NewWriter returns a new Writer.
// Writes to the returned writer are compressed and written to w.
//
// It is the caller's responsibility to call Close on the WriteCloser when done.
// Writes may be buffered and not flushed until Close.
//
// Callers that wish to set the fields in Writer.Header must do so before
// the first call to Write or Close. The Comment and Name header fields are
// UTF-8 strings in Go, but the underlying format requires NUL-terminated ISO
// 8859-1 (Latin-1). NUL or non-Latin-1 runes in those strings will lead to an
// error on Write.
func NewWriter(w io.Writer) *Writer {
z, _ := NewWriterLevel(w, DefaultCompression)
return z
}
// NewWriterLevel is like NewWriter but specifies the compression level instead
// of assuming DefaultCompression.
//
// The compression level can be ConstantCompression, DefaultCompression,
// NoCompression, or any integer value between BestSpeed and BestCompression
// inclusive. The error returned will be nil if the level is valid.
func NewWriterLevel(w io.Writer, level int) (*Writer, error) {
if level < ConstantCompression || level > BestCompression {
return nil, fmt.Errorf("gzip: invalid compression level: %d", level)
}
z := new(Writer)
z.init(w, level)
return z, nil
}
func (z *Writer) init(w io.Writer, level int) {
digest := z.digest
if digest != nil {
digest.Reset()
} else {
digest = crc32.NewIEEE()
}
compressor := z.compressor
if compressor != nil {
compressor.Reset(w)
}
*z = Writer{
Header: Header{
OS: 255, // unknown
},
w: w,
level: level,
digest: digest,
compressor: compressor,
}
}
// Reset discards the Writer z's state and makes it equivalent to the
// result of its original state from NewWriter or NewWriterLevel, but
// writing to w instead. This permits reusing a Writer rather than
// allocating a new one.
func (z *Writer) Reset(w io.Writer) {
z.init(w, z.level)
}
// GZIP (RFC 1952) is little-endian, unlike ZLIB (RFC 1950).
func put2(p []byte, v uint16) {
p[0] = uint8(v >> 0)
p[1] = uint8(v >> 8)
}
func put4(p []byte, v uint32) {
p[0] = uint8(v >> 0)
p[1] = uint8(v >> 8)
p[2] = uint8(v >> 16)
p[3] = uint8(v >> 24)
}
// writeBytes writes a length-prefixed byte slice to z.w.
func (z *Writer) writeBytes(b []byte) error {
if len(b) > 0xffff {
return errors.New("gzip.Write: Extra data is too large")
}
put2(z.buf[0:2], uint16(len(b)))
_, err := z.w.Write(z.buf[0:2])
if err != nil {
return err
}
_, err = z.w.Write(b)
return err
}
// writeString writes a UTF-8 string s in GZIP's format to z.w.
// GZIP (RFC 1952) specifies that strings are NUL-terminated ISO 8859-1 (Latin-1).
func (z *Writer) writeString(s string) (err error) {
// GZIP stores Latin-1 strings; error if non-Latin-1; convert if non-ASCII.
needconv := false
for _, v := range s {
if v == 0 || v > 0xff {
return errors.New("gzip.Write: non-Latin-1 header string")
}
if v > 0x7f {
needconv = true
}
}
if needconv {
b := make([]byte, 0, len(s))
for _, v := range s {
b = append(b, byte(v))
}
_, err = z.w.Write(b)
} else {
_, err = io.WriteString(z.w, s)
}
if err != nil {
return err
}
// GZIP strings are NUL-terminated.
z.buf[0] = 0
_, err = z.w.Write(z.buf[0:1])
return err
}
// Write writes a compressed form of p to the underlying io.Writer. The
// compressed bytes are not necessarily flushed until the Writer is closed.
func (z *Writer) Write(p []byte) (int, error) {
if z.err != nil {
return 0, z.err
}
var n int
// Write the GZIP header lazily.
if !z.wroteHeader {
z.wroteHeader = true
z.buf[0] = gzipID1
z.buf[1] = gzipID2
z.buf[2] = gzipDeflate
z.buf[3] = 0
if z.Extra != nil {
z.buf[3] |= 0x04
}
if z.Name != "" {
z.buf[3] |= 0x08
}
if z.Comment != "" {
z.buf[3] |= 0x10
}
put4(z.buf[4:8], uint32(z.ModTime.Unix()))
if z.level == BestCompression {
z.buf[8] = 2
} else if z.level == BestSpeed {
z.buf[8] = 4
} else {
z.buf[8] = 0
}
z.buf[9] = z.OS
n, z.err = z.w.Write(z.buf[0:10])
if z.err != nil {
return n, z.err
}
if z.Extra != nil {
z.err = z.writeBytes(z.Extra)
if z.err != nil {
return n, z.err
}
}
if z.Name != "" {
z.err = z.writeString(z.Name)
if z.err != nil {
return n, z.err
}
}
if z.Comment != "" {
z.err = z.writeString(z.Comment)
if z.err != nil {
return n, z.err
}
}
if z.compressor == nil {
z.compressor, _ = flate.NewWriter(z.w, z.level)
}
}
z.size += uint32(len(p))
z.digest.Write(p)
n, z.err = z.compressor.Write(p)
return n, z.err
}
// Flush flushes any pending compressed data to the underlying writer.
//
// It is useful mainly in compressed network protocols, to ensure that
// a remote reader has enough data to reconstruct a packet. Flush does
// not return until the data has been written. If the underlying
// writer returns an error, Flush returns that error.
//
// In the terminology of the zlib library, Flush is equivalent to Z_SYNC_FLUSH.
func (z *Writer) Flush() error {
if z.err != nil {
return z.err
}
if z.closed {
return nil
}
if !z.wroteHeader {
z.Write(nil)
if z.err != nil {
return z.err
}
}
z.err = z.compressor.Flush()
return z.err
}
// Close closes the Writer, flushing any unwritten data to the underlying
// io.Writer, but does not close the underlying io.Writer.
func (z *Writer) Close() error {
if z.err != nil {
return z.err
}
if z.closed {
return nil
}
z.closed = true
if !z.wroteHeader {
z.Write(nil)
if z.err != nil {
return z.err
}
}
z.err = z.compressor.Close()
if z.err != nil {
return z.err
}
put4(z.buf[0:4], z.digest.Sum32())
put4(z.buf[4:8], z.size)
_, z.err = z.w.Write(z.buf[0:8])
return z.err
}

519
vendor/github.com/klauspost/compress/gzip/gzip_test.go generated vendored Normal file
View File

@@ -0,0 +1,519 @@
// Copyright 2010 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package gzip
import (
"bufio"
"bytes"
oldgz "compress/gzip"
"io"
"io/ioutil"
"math/rand"
"testing"
"time"
)
// TestEmpty tests that an empty payload still forms a valid GZIP stream.
func TestEmpty(t *testing.T) {
buf := new(bytes.Buffer)
if err := NewWriter(buf).Close(); err != nil {
t.Fatalf("Writer.Close: %v", err)
}
r, err := NewReader(buf)
if err != nil {
t.Fatalf("NewReader: %v", err)
}
b, err := ioutil.ReadAll(r)
if err != nil {
t.Fatalf("ReadAll: %v", err)
}
if len(b) != 0 {
t.Fatalf("got %d bytes, want 0", len(b))
}
if err := r.Close(); err != nil {
t.Fatalf("Reader.Close: %v", err)
}
}
// TestRoundTrip tests that gzipping and then gunzipping is the identity
// function.
func TestRoundTrip(t *testing.T) {
buf := new(bytes.Buffer)
w := NewWriter(buf)
w.Comment = "comment"
w.Extra = []byte("extra")
w.ModTime = time.Unix(1e8, 0)
w.Name = "name"
if _, err := w.Write([]byte("payload")); err != nil {
t.Fatalf("Write: %v", err)
}
if err := w.Close(); err != nil {
t.Fatalf("Writer.Close: %v", err)
}
r, err := NewReader(buf)
if err != nil {
t.Fatalf("NewReader: %v", err)
}
b, err := ioutil.ReadAll(r)
if err != nil {
t.Fatalf("ReadAll: %v", err)
}
if string(b) != "payload" {
t.Fatalf("payload is %q, want %q", string(b), "payload")
}
if r.Comment != "comment" {
t.Fatalf("comment is %q, want %q", r.Comment, "comment")
}
if string(r.Extra) != "extra" {
t.Fatalf("extra is %q, want %q", r.Extra, "extra")
}
if r.ModTime.Unix() != 1e8 {
t.Fatalf("mtime is %d, want %d", r.ModTime.Unix(), uint32(1e8))
}
if r.Name != "name" {
t.Fatalf("name is %q, want %q", r.Name, "name")
}
if err := r.Close(); err != nil {
t.Fatalf("Reader.Close: %v", err)
}
}
// TestLatin1 tests the internal functions for converting to and from Latin-1.
func TestLatin1(t *testing.T) {
latin1 := []byte{0xc4, 'u', 0xdf, 'e', 'r', 'u', 'n', 'g', 0}
utf8 := "Äußerung"
z := Reader{r: bufio.NewReader(bytes.NewReader(latin1))}
s, err := z.readString()
if err != nil {
t.Fatalf("readString: %v", err)
}
if s != utf8 {
t.Fatalf("read latin-1: got %q, want %q", s, utf8)
}
buf := bytes.NewBuffer(make([]byte, 0, len(latin1)))
c := Writer{w: buf}
if err = c.writeString(utf8); err != nil {
t.Fatalf("writeString: %v", err)
}
s = buf.String()
if s != string(latin1) {
t.Fatalf("write utf-8: got %q, want %q", s, string(latin1))
}
}
// TestLatin1RoundTrip tests that metadata that is representable in Latin-1
// survives a round trip.
func TestLatin1RoundTrip(t *testing.T) {
testCases := []struct {
name string
ok bool
}{
{"", true},
{"ASCII is OK", true},
{"unless it contains a NUL\x00", false},
{"no matter where \x00 occurs", false},
{"\x00\x00\x00", false},
{"Látin-1 also passes (U+00E1)", true},
{"but LĀtin Extended-A (U+0100) does not", false},
{"neither does 日本語", false},
{"invalid UTF-8 also \xffails", false},
{"\x00 as does Látin-1 with NUL", false},
}
for _, tc := range testCases {
buf := new(bytes.Buffer)
w := NewWriter(buf)
w.Name = tc.name
err := w.Close()
if (err == nil) != tc.ok {
t.Errorf("Writer.Close: name = %q, err = %v", tc.name, err)
continue
}
if !tc.ok {
continue
}
r, err := NewReader(buf)
if err != nil {
t.Errorf("NewReader: %v", err)
continue
}
_, err = ioutil.ReadAll(r)
if err != nil {
t.Errorf("ReadAll: %v", err)
continue
}
if r.Name != tc.name {
t.Errorf("name is %q, want %q", r.Name, tc.name)
continue
}
if err := r.Close(); err != nil {
t.Errorf("Reader.Close: %v", err)
continue
}
}
}
func TestWriterFlush(t *testing.T) {
buf := new(bytes.Buffer)
w := NewWriter(buf)
w.Comment = "comment"
w.Extra = []byte("extra")
w.ModTime = time.Unix(1e8, 0)
w.Name = "name"
n0 := buf.Len()
if n0 != 0 {
t.Fatalf("buffer size = %d before writes; want 0", n0)
}
if err := w.Flush(); err != nil {
t.Fatal(err)
}
n1 := buf.Len()
if n1 == 0 {
t.Fatal("no data after first flush")
}
w.Write([]byte("x"))
n2 := buf.Len()
if n1 != n2 {
t.Fatalf("after writing a single byte, size changed from %d to %d; want no change", n1, n2)
}
if err := w.Flush(); err != nil {
t.Fatal(err)
}
n3 := buf.Len()
if n2 == n3 {
t.Fatal("Flush didn't flush any data")
}
}
// Multiple gzip files concatenated form a valid gzip file.
func TestConcat(t *testing.T) {
var buf bytes.Buffer
w := NewWriter(&buf)
w.Write([]byte("hello "))
w.Close()
w = NewWriter(&buf)
w.Write([]byte("world\n"))
w.Close()
r, err := NewReader(&buf)
data, err := ioutil.ReadAll(r)
if string(data) != "hello world\n" || err != nil {
t.Fatalf("ReadAll = %q, %v, want %q, nil", data, err, "hello world")
}
}
func TestWriterReset(t *testing.T) {
buf := new(bytes.Buffer)
buf2 := new(bytes.Buffer)
z := NewWriter(buf)
msg := []byte("hello world")
z.Write(msg)
z.Close()
z.Reset(buf2)
z.Write(msg)
z.Close()
if buf.String() != buf2.String() {
t.Errorf("buf2 %q != original buf of %q", buf2.String(), buf.String())
}
}
var testbuf []byte
func testFile(i, level int, t *testing.T) {
dat, _ := ioutil.ReadFile("testdata/test.json")
dl := len(dat)
if len(testbuf) != i*dl {
// Make results predictable
testbuf = make([]byte, i*dl)
for j := 0; j < i; j++ {
copy(testbuf[j*dl:j*dl+dl], dat)
}
}
br := bytes.NewBuffer(testbuf)
var buf bytes.Buffer
w, err := NewWriterLevel(&buf, DefaultCompression)
if err != nil {
t.Fatal(err)
}
n, err := io.Copy(w, br)
if err != nil {
t.Fatal(err)
}
if int(n) != len(testbuf) {
t.Fatal("Short write:", n, "!=", testbuf)
}
err = w.Close()
if err != nil {
t.Fatal(err)
}
r, err := NewReader(&buf)
if err != nil {
t.Fatal(err.Error())
}
decoded, err := ioutil.ReadAll(r)
if err != nil {
t.Fatal(err.Error())
}
if !bytes.Equal(testbuf, decoded) {
t.Errorf("decoded content does not match.")
}
}
func TestFile1xM2(t *testing.T) { testFile(1, -2, t) }
func TestFile1xM1(t *testing.T) { testFile(1, -1, t) }
func TestFile1x0(t *testing.T) { testFile(1, 0, t) }
func TestFile1x1(t *testing.T) { testFile(1, 1, t) }
func TestFile1x2(t *testing.T) { testFile(1, 2, t) }
func TestFile1x3(t *testing.T) { testFile(1, 3, t) }
func TestFile1x4(t *testing.T) { testFile(1, 4, t) }
func TestFile1x5(t *testing.T) { testFile(1, 5, t) }
func TestFile1x6(t *testing.T) { testFile(1, 6, t) }
func TestFile1x7(t *testing.T) { testFile(1, 7, t) }
func TestFile1x8(t *testing.T) { testFile(1, 8, t) }
func TestFile1x9(t *testing.T) { testFile(1, 9, t) }
func TestFile10(t *testing.T) { testFile(10, DefaultCompression, t) }
func TestFile50(t *testing.T) {
if testing.Short() {
t.Skip("skipping during short test")
}
testFile(50, DefaultCompression, t)
}
func TestFile200(t *testing.T) {
if testing.Short() {
t.Skip("skipping during short test")
}
testFile(200, BestSpeed, t)
}
func testBigGzip(i int, t *testing.T) {
if len(testbuf) != i {
// Make results predictable
rand.Seed(1337)
testbuf = make([]byte, i)
for idx := range testbuf {
testbuf[idx] = byte(65 + rand.Intn(20))
}
}
c := BestCompression
if testing.Short() {
c = BestSpeed
}
br := bytes.NewBuffer(testbuf)
var buf bytes.Buffer
w, err := NewWriterLevel(&buf, c)
if err != nil {
t.Fatal(err)
}
n, err := io.Copy(w, br)
if err != nil {
t.Fatal(err)
}
if int(n) != len(testbuf) {
t.Fatal("Short write:", n, "!=", len(testbuf))
}
err = w.Close()
if err != nil {
t.Fatal(err.Error())
}
r, err := NewReader(&buf)
if err != nil {
t.Fatal(err.Error())
}
decoded, err := ioutil.ReadAll(r)
if err != nil {
t.Fatal(err.Error())
}
if !bytes.Equal(testbuf, decoded) {
t.Errorf("decoded content does not match.")
}
}
func TestGzip1K(t *testing.T) { testBigGzip(1000, t) }
func TestGzip100K(t *testing.T) { testBigGzip(100000, t) }
func TestGzip1M(t *testing.T) {
if testing.Short() {
t.Skip("skipping during short test")
}
testBigGzip(1000000, t)
}
func TestGzip10M(t *testing.T) {
if testing.Short() {
t.Skip("skipping during short test")
}
testBigGzip(10000000, t)
}
// Test if two runs produce identical results.
func TestDeterministicLM2(t *testing.T) { testDeterm(-2, t) }
// Level 0 is not deterministic since it depends on the size of each write.
// func TestDeterministicL0(t *testing.T) { testDeterm(0, t) }
func TestDeterministicL1(t *testing.T) { testDeterm(1, t) }
func TestDeterministicL2(t *testing.T) { testDeterm(2, t) }
func TestDeterministicL3(t *testing.T) { testDeterm(3, t) }
func TestDeterministicL4(t *testing.T) { testDeterm(4, t) }
func TestDeterministicL5(t *testing.T) { testDeterm(5, t) }
func TestDeterministicL6(t *testing.T) { testDeterm(6, t) }
func TestDeterministicL7(t *testing.T) { testDeterm(7, t) }
func TestDeterministicL8(t *testing.T) { testDeterm(8, t) }
func TestDeterministicL9(t *testing.T) { testDeterm(9, t) }
func testDeterm(i int, t *testing.T) {
var length = 500000
if testing.Short() {
length = 100000
}
rand.Seed(1337)
t1 := make([]byte, length)
for idx := range t1 {
t1[idx] = byte(65 + rand.Intn(8))
}
br := bytes.NewBuffer(t1)
var b1 bytes.Buffer
w, err := NewWriterLevel(&b1, i)
if err != nil {
t.Fatal(err)
}
_, err = io.Copy(w, br)
if err != nil {
t.Fatal(err)
}
w.Flush()
w.Close()
// We recreate the buffer, so we have a goos chance of getting a
// different memory address.
rand.Seed(1337)
t2 := make([]byte, length)
for idx := range t2 {
t2[idx] = byte(65 + rand.Intn(8))
}
br2 := bytes.NewBuffer(t2)
var b2 bytes.Buffer
w2, err := NewWriterLevel(&b2, i)
if err != nil {
t.Fatal(err)
}
// We write the same data, but with a different size than
// the default copy.
for {
_, err = io.CopyN(w2, br2, 1234)
if err == io.EOF {
err = nil
break
} else if err != nil {
break
}
}
if err != nil {
t.Fatal(err)
}
w2.Flush()
w2.Close()
b1b := b1.Bytes()
b2b := b2.Bytes()
if bytes.Compare(b1b, b2b) != 0 {
t.Fatalf("Level %d did not produce deterministric result, len(a) = %d, len(b) = %d", i, len(b1b), len(b2b))
}
}
func BenchmarkGzipLM2(b *testing.B) { benchmarkGzipN(b, -2) }
func BenchmarkGzipL1(b *testing.B) { benchmarkGzipN(b, 1) }
func BenchmarkGzipL2(b *testing.B) { benchmarkGzipN(b, 2) }
func BenchmarkGzipL3(b *testing.B) { benchmarkGzipN(b, 3) }
func BenchmarkGzipL4(b *testing.B) { benchmarkGzipN(b, 4) }
func BenchmarkGzipL5(b *testing.B) { benchmarkGzipN(b, 5) }
func BenchmarkGzipL6(b *testing.B) { benchmarkGzipN(b, 6) }
func BenchmarkGzipL7(b *testing.B) { benchmarkGzipN(b, 7) }
func BenchmarkGzipL8(b *testing.B) { benchmarkGzipN(b, 8) }
func BenchmarkGzipL9(b *testing.B) { benchmarkGzipN(b, 9) }
func benchmarkGzipN(b *testing.B, level int) {
dat, _ := ioutil.ReadFile("testdata/test.json")
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
b.SetBytes(int64(len(dat)))
w, _ := NewWriterLevel(ioutil.Discard, level)
b.ResetTimer()
for n := 0; n < b.N; n++ {
w.Reset(ioutil.Discard)
n, err := w.Write(dat)
if n != len(dat) {
panic("short write")
}
if err != nil {
panic(err)
}
err = w.Close()
if err != nil {
panic(err)
}
}
}
func BenchmarkOldGzipL1(b *testing.B) { benchmarkOldGzipN(b, 1) }
func BenchmarkOldGzipL2(b *testing.B) { benchmarkOldGzipN(b, 2) }
func BenchmarkOldGzipL3(b *testing.B) { benchmarkOldGzipN(b, 3) }
func BenchmarkOldGzipL4(b *testing.B) { benchmarkOldGzipN(b, 4) }
func BenchmarkOldGzipL5(b *testing.B) { benchmarkOldGzipN(b, 5) }
func BenchmarkOldGzipL6(b *testing.B) { benchmarkOldGzipN(b, 6) }
func BenchmarkOldGzipL7(b *testing.B) { benchmarkOldGzipN(b, 7) }
func BenchmarkOldGzipL8(b *testing.B) { benchmarkOldGzipN(b, 8) }
func BenchmarkOldGzipL9(b *testing.B) { benchmarkOldGzipN(b, 9) }
func benchmarkOldGzipN(b *testing.B, level int) {
dat, _ := ioutil.ReadFile("testdata/test.json")
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
dat = append(dat, dat...)
b.SetBytes(int64(len(dat)))
w, _ := oldgz.NewWriterLevel(ioutil.Discard, level)
b.ResetTimer()
for n := 0; n < b.N; n++ {
w.Reset(ioutil.Discard)
n, err := w.Write(dat)
if n != len(dat) {
panic("short write")
}
if err != nil {
panic(err)
}
err = w.Close()
if err != nil {
panic(err)
}
}
}

BIN
vendor/github.com/klauspost/compress/gzip/testdata/issue6550.gz generated vendored Normal file
View File

Binary file not shown.

5902
vendor/github.com/klauspost/compress/gzip/testdata/test.json generated vendored Normal file
View File

File diff suppressed because it is too large Load Diff

8472
vendor/github.com/klauspost/compress/testdata/Mark.Twain-Tom.Sawyer.txt generated vendored Normal file
View File

File diff suppressed because it is too large Load Diff

1
vendor/github.com/klauspost/compress/testdata/e.txt generated vendored Normal file
View File

File diff suppressed because one or more lines are too long

1
vendor/github.com/klauspost/compress/testdata/pi.txt generated vendored Normal file
View File

File diff suppressed because one or more lines are too long

22
vendor/github.com/klauspost/cpuid/LICENSE generated vendored Normal file
View File

@@ -0,0 +1,22 @@
The MIT License (MIT)
Copyright (c) 2015 Klaus Post
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

145
vendor/github.com/klauspost/cpuid/README.md generated vendored Normal file
View File

@@ -0,0 +1,145 @@
# cpuid
Package cpuid provides information about the CPU running the current program.
CPU features are detected on startup, and kept for fast access through the life of the application.
Currently x86 / x64 (AMD64) is supported, and no external C (cgo) code is used, which should make the library very easy to use.
You can access the CPU information by accessing the shared CPU variable of the cpuid library.
Package home: https://github.com/klauspost/cpuid
[![GoDoc][1]][2] [![Build Status][3]][4]
[1]: https://godoc.org/github.com/klauspost/cpuid?status.svg
[2]: https://godoc.org/github.com/klauspost/cpuid
[3]: https://travis-ci.org/klauspost/cpuid.svg
[4]: https://travis-ci.org/klauspost/cpuid
# features
## CPU Instructions
* **CMOV** (i686 CMOV)
* **NX** (NX (No-Execute) bit)
* **AMD3DNOW** (AMD 3DNOW)
* **AMD3DNOWEXT** (AMD 3DNowExt)
* **MMX** (standard MMX)
* **MMXEXT** (SSE integer functions or AMD MMX ext)
* **SSE** (SSE functions)
* **SSE2** (P4 SSE functions)
* **SSE3** (Prescott SSE3 functions)
* **SSSE3** (Conroe SSSE3 functions)
* **SSE4** (Penryn SSE4.1 functions)
* **SSE4A** (AMD Barcelona microarchitecture SSE4a instructions)
* **SSE42** (Nehalem SSE4.2 functions)
* **AVX** (AVX functions)
* **AVX2** (AVX2 functions)
* **FMA3** (Intel FMA 3)
* **FMA4** (Bulldozer FMA4 functions)
* **XOP** (Bulldozer XOP functions)
* **F16C** (Half-precision floating-point conversion)
* **BMI1** (Bit Manipulation Instruction Set 1)
* **BMI2** (Bit Manipulation Instruction Set 2)
* **TBM** (AMD Trailing Bit Manipulation)
* **LZCNT** (LZCNT instruction)
* **POPCNT** (POPCNT instruction)
* **AESNI** (Advanced Encryption Standard New Instructions)
* **CLMUL** (Carry-less Multiplication)
* **HTT** (Hyperthreading (enabled))
* **HLE** (Hardware Lock Elision)
* **RTM** (Restricted Transactional Memory)
* **RDRAND** (RDRAND instruction is available)
* **RDSEED** (RDSEED instruction is available)
* **ADX** (Intel ADX (Multi-Precision Add-Carry Instruction Extensions))
* **SHA** (Intel SHA Extensions)
* **AVX512F** (AVX-512 Foundation)
* **AVX512DQ** (AVX-512 Doubleword and Quadword Instructions)
* **AVX512IFMA** (AVX-512 Integer Fused Multiply-Add Instructions)
* **AVX512PF** (AVX-512 Prefetch Instructions)
* **AVX512ER** (AVX-512 Exponential and Reciprocal Instructions)
* **AVX512CD** (AVX-512 Conflict Detection Instructions)
* **AVX512BW** (AVX-512 Byte and Word Instructions)
* **AVX512VL** (AVX-512 Vector Length Extensions)
* **AVX512VBMI** (AVX-512 Vector Bit Manipulation Instructions)
* **MPX** (Intel MPX (Memory Protection Extensions))
* **ERMS** (Enhanced REP MOVSB/STOSB)
* **RDTSCP** (RDTSCP Instruction)
* **CX16** (CMPXCHG16B Instruction)
* **SGX** (Software Guard Extensions, with activation details)
## Performance
* **RDTSCP()** Returns current cycle count. Can be used for benchmarking.
* **SSE2SLOW** (SSE2 is supported, but usually not faster)
* **SSE3SLOW** (SSE3 is supported, but usually not faster)
* **ATOM** (Atom processor, some SSSE3 instructions are slower)
* **Cache line** (Probable size of a cache line).
* **L1, L2, L3 Cache size** on newer Intel/AMD CPUs.
## Cpu Vendor/VM
* **Intel**
* **AMD**
* **VIA**
* **Transmeta**
* **NSC**
* **KVM** (Kernel-based Virtual Machine)
* **MSVM** (Microsoft Hyper-V or Windows Virtual PC)
* **VMware**
* **XenHVM**
# installing
```go get github.com/klauspost/cpuid```
# example
```Go
package main
import (
"fmt"
"github.com/klauspost/cpuid"
)
func main() {
// Print basic CPU information:
fmt.Println("Name:", cpuid.CPU.BrandName)
fmt.Println("PhysicalCores:", cpuid.CPU.PhysicalCores)
fmt.Println("ThreadsPerCore:", cpuid.CPU.ThreadsPerCore)
fmt.Println("LogicalCores:", cpuid.CPU.LogicalCores)
fmt.Println("Family", cpuid.CPU.Family, "Model:", cpuid.CPU.Model)
fmt.Println("Features:", cpuid.CPU.Features)
fmt.Println("Cacheline bytes:", cpuid.CPU.CacheLine)
fmt.Println("L1 Data Cache:", cpuid.CPU.Cache.L1D, "bytes")
fmt.Println("L1 Instruction Cache:", cpuid.CPU.Cache.L1D, "bytes")
fmt.Println("L2 Cache:", cpuid.CPU.Cache.L2, "bytes")
fmt.Println("L3 Cache:", cpuid.CPU.Cache.L3, "bytes")
// Test if we have a specific feature:
if cpuid.CPU.SSE() {
fmt.Println("We have Streaming SIMD Extensions")
}
}
```
Sample output:
```
>go run main.go
Name: Intel(R) Core(TM) i5-2540M CPU @ 2.60GHz
PhysicalCores: 2
ThreadsPerCore: 2
LogicalCores: 4
Family 6 Model: 42
Features: CMOV,MMX,MMXEXT,SSE,SSE2,SSE3,SSSE3,SSE4.1,SSE4.2,AVX,AESNI,CLMUL
Cacheline bytes: 64
We have Streaming SIMD Extensions
```
# private package
In the "private" folder you can find an autogenerated version of the library you can include in your own packages.
For this purpose all exports are removed, and functions and constants are lowercased.
This is not a recommended way of using the library, but provided for convenience, if it is difficult for you to use external packages.
# license
This code is published under an MIT license. See LICENSE file for more information.

1022
vendor/github.com/klauspost/cpuid/cpuid.go generated vendored Normal file
View File

File diff suppressed because it is too large Load Diff

42
vendor/github.com/klauspost/cpuid/cpuid_386.s generated vendored Normal file
View File

@@ -0,0 +1,42 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build 386,!gccgo
// func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuid(SB), 7, $0
XORL CX, CX
MOVL op+0(FP), AX
CPUID
MOVL AX, eax+4(FP)
MOVL BX, ebx+8(FP)
MOVL CX, ecx+12(FP)
MOVL DX, edx+16(FP)
RET
// func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuidex(SB), 7, $0
MOVL op+0(FP), AX
MOVL op2+4(FP), CX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func xgetbv(index uint32) (eax, edx uint32)
TEXT ·asmXgetbv(SB), 7, $0
MOVL index+0(FP), CX
BYTE $0x0f; BYTE $0x01; BYTE $0xd0 // XGETBV
MOVL AX, eax+4(FP)
MOVL DX, edx+8(FP)
RET
// func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
TEXT ·asmRdtscpAsm(SB), 7, $0
BYTE $0x0F; BYTE $0x01; BYTE $0xF9 // RDTSCP
MOVL AX, eax+0(FP)
MOVL BX, ebx+4(FP)
MOVL CX, ecx+8(FP)
MOVL DX, edx+12(FP)
RET

42
vendor/github.com/klauspost/cpuid/cpuid_amd64.s generated vendored Normal file
View File

@@ -0,0 +1,42 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
//+build amd64,!gccgo
// func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuid(SB), 7, $0
XORQ CX, CX
MOVL op+0(FP), AX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuidex(SB), 7, $0
MOVL op+0(FP), AX
MOVL op2+4(FP), CX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func asmXgetbv(index uint32) (eax, edx uint32)
TEXT ·asmXgetbv(SB), 7, $0
MOVL index+0(FP), CX
BYTE $0x0f; BYTE $0x01; BYTE $0xd0 // XGETBV
MOVL AX, eax+8(FP)
MOVL DX, edx+12(FP)
RET
// func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
TEXT ·asmRdtscpAsm(SB), 7, $0
BYTE $0x0F; BYTE $0x01; BYTE $0xF9 // RDTSCP
MOVL AX, eax+0(FP)
MOVL BX, ebx+4(FP)
MOVL CX, ecx+8(FP)
MOVL DX, edx+12(FP)
RET

727
vendor/github.com/klauspost/cpuid/cpuid_test.go generated vendored Normal file
View File

@@ -0,0 +1,727 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
package cpuid
import (
"fmt"
"testing"
)
// There is no real way to test a CPU identifier, since results will
// obviously differ on each machine.
func TestCPUID(t *testing.T) {
n := maxFunctionID()
t.Logf("Max Function:0x%x\n", n)
n = maxExtendedFunction()
t.Logf("Max Extended Function:0x%x\n", n)
t.Log("Name:", CPU.BrandName)
t.Log("PhysicalCores:", CPU.PhysicalCores)
t.Log("ThreadsPerCore:", CPU.ThreadsPerCore)
t.Log("LogicalCores:", CPU.LogicalCores)
t.Log("Family", CPU.Family, "Model:", CPU.Model)
t.Log("Features:", CPU.Features)
t.Log("Cacheline bytes:", CPU.CacheLine)
t.Log("L1 Instruction Cache:", CPU.Cache.L1I, "bytes")
t.Log("L1 Data Cache:", CPU.Cache.L1D, "bytes")
t.Log("L2 Cache:", CPU.Cache.L2, "bytes")
t.Log("L3 Cache:", CPU.Cache.L3, "bytes")
if CPU.SSE2() {
t.Log("We have SSE2")
}
}
func TestDumpCPUID(t *testing.T) {
n := int(maxFunctionID())
for i := 0; i <= n; i++ {
a, b, c, d := cpuidex(uint32(i), 0)
t.Logf("CPUID %08x: %08x-%08x-%08x-%08x", i, a, b, c, d)
ex := uint32(1)
for {
a2, b2, c2, d2 := cpuidex(uint32(i), ex)
if a2 == a && b2 == b && d2 == d || ex > 50 || a2 == 0 {
break
}
t.Logf("CPUID %08x: %08x-%08x-%08x-%08x", i, a2, b2, c2, d2)
a, b, c, d = a2, b2, c2, d2
ex++
}
}
n2 := maxExtendedFunction()
for i := uint32(0x80000000); i <= n2; i++ {
a, b, c, d := cpuid(i)
t.Logf("CPUID %08x: %08x-%08x-%08x-%08x", i, a, b, c, d)
}
}
func Example() {
// Print basic CPU information:
fmt.Println("Name:", CPU.BrandName)
fmt.Println("PhysicalCores:", CPU.PhysicalCores)
fmt.Println("ThreadsPerCore:", CPU.ThreadsPerCore)
fmt.Println("LogicalCores:", CPU.LogicalCores)
fmt.Println("Family", CPU.Family, "Model:", CPU.Model)
fmt.Println("Features:", CPU.Features)
fmt.Println("Cacheline bytes:", CPU.CacheLine)
// Test if we have a specific feature:
if CPU.SSE() {
fmt.Println("We have Streaming SIMD Extensions")
}
}
func TestBrandNameZero(t *testing.T) {
if len(CPU.BrandName) > 0 {
// Cut out last byte
last := []byte(CPU.BrandName[len(CPU.BrandName)-1:])
if last[0] == 0 {
t.Fatal("last byte was zero")
} else if last[0] == 32 {
t.Fatal("whitespace wasn't trimmed")
}
}
}
// Generated here: http://play.golang.org/p/mko-0tFt0Q
// TestCmov tests Cmov() function
func TestCmov(t *testing.T) {
got := CPU.Cmov()
expected := CPU.Features&CMOV == CMOV
if got != expected {
t.Fatalf("Cmov: expected %v, got %v", expected, got)
}
t.Log("CMOV Support:", got)
}
// TestAmd3dnow tests Amd3dnow() function
func TestAmd3dnow(t *testing.T) {
got := CPU.Amd3dnow()
expected := CPU.Features&AMD3DNOW == AMD3DNOW
if got != expected {
t.Fatalf("Amd3dnow: expected %v, got %v", expected, got)
}
t.Log("AMD3DNOW Support:", got)
}
// TestAmd3dnowExt tests Amd3dnowExt() function
func TestAmd3dnowExt(t *testing.T) {
got := CPU.Amd3dnowExt()
expected := CPU.Features&AMD3DNOWEXT == AMD3DNOWEXT
if got != expected {
t.Fatalf("Amd3dnowExt: expected %v, got %v", expected, got)
}
t.Log("AMD3DNOWEXT Support:", got)
}
// TestMMX tests MMX() function
func TestMMX(t *testing.T) {
got := CPU.MMX()
expected := CPU.Features&MMX == MMX
if got != expected {
t.Fatalf("MMX: expected %v, got %v", expected, got)
}
t.Log("MMX Support:", got)
}
// TestMMXext tests MMXext() function
func TestMMXext(t *testing.T) {
got := CPU.MMXExt()
expected := CPU.Features&MMXEXT == MMXEXT
if got != expected {
t.Fatalf("MMXExt: expected %v, got %v", expected, got)
}
t.Log("MMXEXT Support:", got)
}
// TestSSE tests SSE() function
func TestSSE(t *testing.T) {
got := CPU.SSE()
expected := CPU.Features&SSE == SSE
if got != expected {
t.Fatalf("SSE: expected %v, got %v", expected, got)
}
t.Log("SSE Support:", got)
}
// TestSSE2 tests SSE2() function
func TestSSE2(t *testing.T) {
got := CPU.SSE2()
expected := CPU.Features&SSE2 == SSE2
if got != expected {
t.Fatalf("SSE2: expected %v, got %v", expected, got)
}
t.Log("SSE2 Support:", got)
}
// TestSSE3 tests SSE3() function
func TestSSE3(t *testing.T) {
got := CPU.SSE3()
expected := CPU.Features&SSE3 == SSE3
if got != expected {
t.Fatalf("SSE3: expected %v, got %v", expected, got)
}
t.Log("SSE3 Support:", got)
}
// TestSSSE3 tests SSSE3() function
func TestSSSE3(t *testing.T) {
got := CPU.SSSE3()
expected := CPU.Features&SSSE3 == SSSE3
if got != expected {
t.Fatalf("SSSE3: expected %v, got %v", expected, got)
}
t.Log("SSSE3 Support:", got)
}
// TestSSE4 tests SSE4() function
func TestSSE4(t *testing.T) {
got := CPU.SSE4()
expected := CPU.Features&SSE4 == SSE4
if got != expected {
t.Fatalf("SSE4: expected %v, got %v", expected, got)
}
t.Log("SSE4 Support:", got)
}
// TestSSE42 tests SSE42() function
func TestSSE42(t *testing.T) {
got := CPU.SSE42()
expected := CPU.Features&SSE42 == SSE42
if got != expected {
t.Fatalf("SSE42: expected %v, got %v", expected, got)
}
t.Log("SSE42 Support:", got)
}
// TestAVX tests AVX() function
func TestAVX(t *testing.T) {
got := CPU.AVX()
expected := CPU.Features&AVX == AVX
if got != expected {
t.Fatalf("AVX: expected %v, got %v", expected, got)
}
t.Log("AVX Support:", got)
}
// TestAVX2 tests AVX2() function
func TestAVX2(t *testing.T) {
got := CPU.AVX2()
expected := CPU.Features&AVX2 == AVX2
if got != expected {
t.Fatalf("AVX2: expected %v, got %v", expected, got)
}
t.Log("AVX2 Support:", got)
}
// TestFMA3 tests FMA3() function
func TestFMA3(t *testing.T) {
got := CPU.FMA3()
expected := CPU.Features&FMA3 == FMA3
if got != expected {
t.Fatalf("FMA3: expected %v, got %v", expected, got)
}
t.Log("FMA3 Support:", got)
}
// TestFMA4 tests FMA4() function
func TestFMA4(t *testing.T) {
got := CPU.FMA4()
expected := CPU.Features&FMA4 == FMA4
if got != expected {
t.Fatalf("FMA4: expected %v, got %v", expected, got)
}
t.Log("FMA4 Support:", got)
}
// TestXOP tests XOP() function
func TestXOP(t *testing.T) {
got := CPU.XOP()
expected := CPU.Features&XOP == XOP
if got != expected {
t.Fatalf("XOP: expected %v, got %v", expected, got)
}
t.Log("XOP Support:", got)
}
// TestF16C tests F16C() function
func TestF16C(t *testing.T) {
got := CPU.F16C()
expected := CPU.Features&F16C == F16C
if got != expected {
t.Fatalf("F16C: expected %v, got %v", expected, got)
}
t.Log("F16C Support:", got)
}
// TestCX16 tests CX16() function
func TestCX16(t *testing.T) {
got := CPU.CX16()
expected := CPU.Features&CX16 == CX16
if got != expected {
t.Fatalf("CX16: expected %v, got %v", expected, got)
}
t.Log("CX16 Support:", got)
}
// TestSGX tests SGX() function
func TestSGX(t *testing.T) {
got := CPU.SGX.Available
expected := CPU.Features&SGX == SGX
if got != expected {
t.Fatalf("SGX: expected %v, got %v", expected, got)
}
t.Log("SGX Support:", got)
}
// TestBMI1 tests BMI1() function
func TestBMI1(t *testing.T) {
got := CPU.BMI1()
expected := CPU.Features&BMI1 == BMI1
if got != expected {
t.Fatalf("BMI1: expected %v, got %v", expected, got)
}
t.Log("BMI1 Support:", got)
}
// TestBMI2 tests BMI2() function
func TestBMI2(t *testing.T) {
got := CPU.BMI2()
expected := CPU.Features&BMI2 == BMI2
if got != expected {
t.Fatalf("BMI2: expected %v, got %v", expected, got)
}
t.Log("BMI2 Support:", got)
}
// TestTBM tests TBM() function
func TestTBM(t *testing.T) {
got := CPU.TBM()
expected := CPU.Features&TBM == TBM
if got != expected {
t.Fatalf("TBM: expected %v, got %v", expected, got)
}
t.Log("TBM Support:", got)
}
// TestLzcnt tests Lzcnt() function
func TestLzcnt(t *testing.T) {
got := CPU.Lzcnt()
expected := CPU.Features&LZCNT == LZCNT
if got != expected {
t.Fatalf("Lzcnt: expected %v, got %v", expected, got)
}
t.Log("LZCNT Support:", got)
}
// TestLzcnt tests Lzcnt() function
func TestPopcnt(t *testing.T) {
got := CPU.Popcnt()
expected := CPU.Features&POPCNT == POPCNT
if got != expected {
t.Fatalf("Popcnt: expected %v, got %v", expected, got)
}
t.Log("POPCNT Support:", got)
}
// TestAesNi tests AesNi() function
func TestAesNi(t *testing.T) {
got := CPU.AesNi()
expected := CPU.Features&AESNI == AESNI
if got != expected {
t.Fatalf("AesNi: expected %v, got %v", expected, got)
}
t.Log("AESNI Support:", got)
}
// TestHTT tests HTT() function
func TestHTT(t *testing.T) {
got := CPU.HTT()
expected := CPU.Features&HTT == HTT
if got != expected {
t.Fatalf("HTT: expected %v, got %v", expected, got)
}
t.Log("HTT Support:", got)
}
// TestClmul tests Clmul() function
func TestClmul(t *testing.T) {
got := CPU.Clmul()
expected := CPU.Features&CLMUL == CLMUL
if got != expected {
t.Fatalf("Clmul: expected %v, got %v", expected, got)
}
t.Log("CLMUL Support:", got)
}
// TestSSE2Slow tests SSE2Slow() function
func TestSSE2Slow(t *testing.T) {
got := CPU.SSE2Slow()
expected := CPU.Features&SSE2SLOW == SSE2SLOW
if got != expected {
t.Fatalf("SSE2Slow: expected %v, got %v", expected, got)
}
t.Log("SSE2SLOW Support:", got)
}
// TestSSE3Slow tests SSE3slow() function
func TestSSE3Slow(t *testing.T) {
got := CPU.SSE3Slow()
expected := CPU.Features&SSE3SLOW == SSE3SLOW
if got != expected {
t.Fatalf("SSE3slow: expected %v, got %v", expected, got)
}
t.Log("SSE3SLOW Support:", got)
}
// TestAtom tests Atom() function
func TestAtom(t *testing.T) {
got := CPU.Atom()
expected := CPU.Features&ATOM == ATOM
if got != expected {
t.Fatalf("Atom: expected %v, got %v", expected, got)
}
t.Log("ATOM Support:", got)
}
// TestNX tests NX() function (NX (No-Execute) bit)
func TestNX(t *testing.T) {
got := CPU.NX()
expected := CPU.Features&NX == NX
if got != expected {
t.Fatalf("NX: expected %v, got %v", expected, got)
}
t.Log("NX Support:", got)
}
// TestSSE4A tests SSE4A() function (AMD Barcelona microarchitecture SSE4a instructions)
func TestSSE4A(t *testing.T) {
got := CPU.SSE4A()
expected := CPU.Features&SSE4A == SSE4A
if got != expected {
t.Fatalf("SSE4A: expected %v, got %v", expected, got)
}
t.Log("SSE4A Support:", got)
}
// TestHLE tests HLE() function (Hardware Lock Elision)
func TestHLE(t *testing.T) {
got := CPU.HLE()
expected := CPU.Features&HLE == HLE
if got != expected {
t.Fatalf("HLE: expected %v, got %v", expected, got)
}
t.Log("HLE Support:", got)
}
// TestRTM tests RTM() function (Restricted Transactional Memory)
func TestRTM(t *testing.T) {
got := CPU.RTM()
expected := CPU.Features&RTM == RTM
if got != expected {
t.Fatalf("RTM: expected %v, got %v", expected, got)
}
t.Log("RTM Support:", got)
}
// TestRdrand tests RDRAND() function (RDRAND instruction is available)
func TestRdrand(t *testing.T) {
got := CPU.Rdrand()
expected := CPU.Features&RDRAND == RDRAND
if got != expected {
t.Fatalf("Rdrand: expected %v, got %v", expected, got)
}
t.Log("Rdrand Support:", got)
}
// TestRdseed tests RDSEED() function (RDSEED instruction is available)
func TestRdseed(t *testing.T) {
got := CPU.Rdseed()
expected := CPU.Features&RDSEED == RDSEED
if got != expected {
t.Fatalf("Rdseed: expected %v, got %v", expected, got)
}
t.Log("Rdseed Support:", got)
}
// TestADX tests ADX() function (Intel ADX (Multi-Precision Add-Carry Instruction Extensions))
func TestADX(t *testing.T) {
got := CPU.ADX()
expected := CPU.Features&ADX == ADX
if got != expected {
t.Fatalf("ADX: expected %v, got %v", expected, got)
}
t.Log("ADX Support:", got)
}
// TestSHA tests SHA() function (Intel SHA Extensions)
func TestSHA(t *testing.T) {
got := CPU.SHA()
expected := CPU.Features&SHA == SHA
if got != expected {
t.Fatalf("SHA: expected %v, got %v", expected, got)
}
t.Log("SHA Support:", got)
}
// TestAVX512F tests AVX512F() function (AVX-512 Foundation)
func TestAVX512F(t *testing.T) {
got := CPU.AVX512F()
expected := CPU.Features&AVX512F == AVX512F
if got != expected {
t.Fatalf("AVX512F: expected %v, got %v", expected, got)
}
t.Log("AVX512F Support:", got)
}
// TestAVX512DQ tests AVX512DQ() function (AVX-512 Doubleword and Quadword Instructions)
func TestAVX512DQ(t *testing.T) {
got := CPU.AVX512DQ()
expected := CPU.Features&AVX512DQ == AVX512DQ
if got != expected {
t.Fatalf("AVX512DQ: expected %v, got %v", expected, got)
}
t.Log("AVX512DQ Support:", got)
}
// TestAVX512IFMA tests AVX512IFMA() function (AVX-512 Integer Fused Multiply-Add Instructions)
func TestAVX512IFMA(t *testing.T) {
got := CPU.AVX512IFMA()
expected := CPU.Features&AVX512IFMA == AVX512IFMA
if got != expected {
t.Fatalf("AVX512IFMA: expected %v, got %v", expected, got)
}
t.Log("AVX512IFMA Support:", got)
}
// TestAVX512PF tests AVX512PF() function (AVX-512 Prefetch Instructions)
func TestAVX512PF(t *testing.T) {
got := CPU.AVX512PF()
expected := CPU.Features&AVX512PF == AVX512PF
if got != expected {
t.Fatalf("AVX512PF: expected %v, got %v", expected, got)
}
t.Log("AVX512PF Support:", got)
}
// TestAVX512ER tests AVX512ER() function (AVX-512 Exponential and Reciprocal Instructions)
func TestAVX512ER(t *testing.T) {
got := CPU.AVX512ER()
expected := CPU.Features&AVX512ER == AVX512ER
if got != expected {
t.Fatalf("AVX512ER: expected %v, got %v", expected, got)
}
t.Log("AVX512ER Support:", got)
}
// TestAVX512CD tests AVX512CD() function (AVX-512 Conflict Detection Instructions)
func TestAVX512CD(t *testing.T) {
got := CPU.AVX512CD()
expected := CPU.Features&AVX512CD == AVX512CD
if got != expected {
t.Fatalf("AVX512CD: expected %v, got %v", expected, got)
}
t.Log("AVX512CD Support:", got)
}
// TestAVX512BW tests AVX512BW() function (AVX-512 Byte and Word Instructions)
func TestAVX512BW(t *testing.T) {
got := CPU.AVX512BW()
expected := CPU.Features&AVX512BW == AVX512BW
if got != expected {
t.Fatalf("AVX512BW: expected %v, got %v", expected, got)
}
t.Log("AVX512BW Support:", got)
}
// TestAVX512VL tests AVX512VL() function (AVX-512 Vector Length Extensions)
func TestAVX512VL(t *testing.T) {
got := CPU.AVX512VL()
expected := CPU.Features&AVX512VL == AVX512VL
if got != expected {
t.Fatalf("AVX512VL: expected %v, got %v", expected, got)
}
t.Log("AVX512VL Support:", got)
}
// TestAVX512VL tests AVX512VBMI() function (AVX-512 Vector Bit Manipulation Instructions)
func TestAVX512VBMI(t *testing.T) {
got := CPU.AVX512VBMI()
expected := CPU.Features&AVX512VBMI == AVX512VBMI
if got != expected {
t.Fatalf("AVX512VBMI: expected %v, got %v", expected, got)
}
t.Log("AVX512VBMI Support:", got)
}
// TestMPX tests MPX() function (Intel MPX (Memory Protection Extensions))
func TestMPX(t *testing.T) {
got := CPU.MPX()
expected := CPU.Features&MPX == MPX
if got != expected {
t.Fatalf("MPX: expected %v, got %v", expected, got)
}
t.Log("MPX Support:", got)
}
// TestERMS tests ERMS() function (Enhanced REP MOVSB/STOSB)
func TestERMS(t *testing.T) {
got := CPU.ERMS()
expected := CPU.Features&ERMS == ERMS
if got != expected {
t.Fatalf("ERMS: expected %v, got %v", expected, got)
}
t.Log("ERMS Support:", got)
}
// TestVendor writes the detected vendor. Will be 0 if unknown
func TestVendor(t *testing.T) {
t.Log("Vendor ID:", CPU.VendorID)
}
// Intel returns true if vendor is recognized as Intel
func TestIntel(t *testing.T) {
got := CPU.Intel()
expected := CPU.VendorID == Intel
if got != expected {
t.Fatalf("TestIntel: expected %v, got %v", expected, got)
}
t.Log("TestIntel:", got)
}
// AMD returns true if vendor is recognized as AMD
func TestAMD(t *testing.T) {
got := CPU.AMD()
expected := CPU.VendorID == AMD
if got != expected {
t.Fatalf("TestAMD: expected %v, got %v", expected, got)
}
t.Log("TestAMD:", got)
}
// Transmeta returns true if vendor is recognized as Transmeta
func TestTransmeta(t *testing.T) {
got := CPU.Transmeta()
expected := CPU.VendorID == Transmeta
if got != expected {
t.Fatalf("TestTransmeta: expected %v, got %v", expected, got)
}
t.Log("TestTransmeta:", got)
}
// NSC returns true if vendor is recognized as National Semiconductor
func TestNSC(t *testing.T) {
got := CPU.NSC()
expected := CPU.VendorID == NSC
if got != expected {
t.Fatalf("TestNSC: expected %v, got %v", expected, got)
}
t.Log("TestNSC:", got)
}
// VIA returns true if vendor is recognized as VIA
func TestVIA(t *testing.T) {
got := CPU.VIA()
expected := CPU.VendorID == VIA
if got != expected {
t.Fatalf("TestVIA: expected %v, got %v", expected, got)
}
t.Log("TestVIA:", got)
}
// Test VM function
func TestVM(t *testing.T) {
t.Log("Vendor ID:", CPU.VM())
}
// Test RTCounter function
func TestRtCounter(t *testing.T) {
a := CPU.RTCounter()
b := CPU.RTCounter()
t.Log("CPU Counter:", a, b, b-a)
}
// Prints the value of Ia32TscAux()
func TestIa32TscAux(t *testing.T) {
ecx := CPU.Ia32TscAux()
t.Logf("Ia32TscAux:0x%x\n", ecx)
if ecx != 0 {
chip := (ecx & 0xFFF000) >> 12
core := ecx & 0xFFF
t.Log("Likely chip, core:", chip, core)
}
}
func TestThreadsPerCoreNZ(t *testing.T) {
if CPU.ThreadsPerCore == 0 {
t.Fatal("threads per core is zero")
}
}
// Prints the value of LogicalCPU()
func TestLogicalCPU(t *testing.T) {
t.Log("Currently executing on cpu:", CPU.LogicalCPU())
}
func TestMaxFunction(t *testing.T) {
expect := maxFunctionID()
if CPU.maxFunc != expect {
t.Fatal("Max function does not match, expected", expect, "but got", CPU.maxFunc)
}
expect = maxExtendedFunction()
if CPU.maxExFunc != expect {
t.Fatal("Max Extended function does not match, expected", expect, "but got", CPU.maxFunc)
}
}
// This example will calculate the chip/core number on Linux
// Linux encodes numa id (<<12) and core id (8bit) into TSC_AUX.
func ExampleCPUInfo_Ia32TscAux(t *testing.T) {
ecx := CPU.Ia32TscAux()
if ecx == 0 {
fmt.Println("Unknown CPU ID")
return
}
chip := (ecx & 0xFFF000) >> 12
core := ecx & 0xFFF
fmt.Println("Chip, Core:", chip, core)
}
/*
func TestPhysical(t *testing.T) {
var test16 = "CPUID 00000000: 0000000d-756e6547-6c65746e-49656e69 \nCPUID 00000001: 000206d7-03200800-1fbee3ff-bfebfbff \nCPUID 00000002: 76035a01-00f0b2ff-00000000-00ca0000 \nCPUID 00000003: 00000000-00000000-00000000-00000000 \nCPUID 00000004: 3c004121-01c0003f-0000003f-00000000 \nCPUID 00000004: 3c004122-01c0003f-0000003f-00000000 \nCPUID 00000004: 3c004143-01c0003f-000001ff-00000000 \nCPUID 00000004: 3c07c163-04c0003f-00003fff-00000006 \nCPUID 00000005: 00000040-00000040-00000003-00021120 \nCPUID 00000006: 00000075-00000002-00000009-00000000 \nCPUID 00000007: 00000000-00000000-00000000-00000000 \nCPUID 00000008: 00000000-00000000-00000000-00000000 \nCPUID 00000009: 00000001-00000000-00000000-00000000 \nCPUID 0000000a: 07300403-00000000-00000000-00000603 \nCPUID 0000000b: 00000000-00000000-00000003-00000003 \nCPUID 0000000b: 00000005-00000010-00000201-00000003 \nCPUID 0000000c: 00000000-00000000-00000000-00000000 \nCPUID 0000000d: 00000007-00000340-00000340-00000000 \nCPUID 0000000d: 00000001-00000000-00000000-00000000 \nCPUID 0000000d: 00000100-00000240-00000000-00000000 \nCPUID 80000000: 80000008-00000000-00000000-00000000 \nCPUID 80000001: 00000000-00000000-00000001-2c100800 \nCPUID 80000002: 20202020-49202020-6c65746e-20295228 \nCPUID 80000003: 6e6f6558-20295228-20555043-322d3545 \nCPUID 80000004: 20303636-20402030-30322e32-007a4847 \nCPUID 80000005: 00000000-00000000-00000000-00000000 \nCPUID 80000006: 00000000-00000000-01006040-00000000 \nCPUID 80000007: 00000000-00000000-00000000-00000100 \nCPUID 80000008: 0000302e-00000000-00000000-00000000"
restore := mockCPU([]byte(test16))
Detect()
t.Log("Name:", CPU.BrandName)
n := maxFunctionID()
t.Logf("Max Function:0x%x\n", n)
n = maxExtendedFunction()
t.Logf("Max Extended Function:0x%x\n", n)
t.Log("PhysicalCores:", CPU.PhysicalCores)
t.Log("ThreadsPerCore:", CPU.ThreadsPerCore)
t.Log("LogicalCores:", CPU.LogicalCores)
t.Log("Family", CPU.Family, "Model:", CPU.Model)
t.Log("Features:", CPU.Features)
t.Log("Cacheline bytes:", CPU.CacheLine)
t.Log("L1 Instruction Cache:", CPU.Cache.L1I, "bytes")
t.Log("L1 Data Cache:", CPU.Cache.L1D, "bytes")
t.Log("L2 Cache:", CPU.Cache.L2, "bytes")
t.Log("L3 Cache:", CPU.Cache.L3, "bytes")
if CPU.LogicalCores > 0 && CPU.PhysicalCores > 0 {
if CPU.LogicalCores != CPU.PhysicalCores*CPU.ThreadsPerCore {
t.Fatalf("Core count mismatch, LogicalCores (%d) != PhysicalCores (%d) * CPU.ThreadsPerCore (%d)",
CPU.LogicalCores, CPU.PhysicalCores, CPU.ThreadsPerCore)
}
}
if CPU.ThreadsPerCore > 1 && !CPU.HTT() {
t.Fatalf("Hyperthreading not detected")
}
if CPU.ThreadsPerCore == 1 && CPU.HTT() {
t.Fatalf("Hyperthreading detected, but only 1 Thread per core")
}
restore()
Detect()
TestCPUID(t)
}
*/

17
vendor/github.com/klauspost/cpuid/detect_intel.go generated vendored Normal file
View File

@@ -0,0 +1,17 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build 386,!gccgo amd64,!gccgo
package cpuid
func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
func asmXgetbv(index uint32) (eax, edx uint32)
func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
func initCPU() {
cpuid = asmCpuid
cpuidex = asmCpuidex
xgetbv = asmXgetbv
rdtscpAsm = asmRdtscpAsm
}

23
vendor/github.com/klauspost/cpuid/detect_ref.go generated vendored Normal file
View File

@@ -0,0 +1,23 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build !amd64,!386 gccgo
package cpuid
func initCPU() {
cpuid = func(op uint32) (eax, ebx, ecx, edx uint32) {
return 0, 0, 0, 0
}
cpuidex = func(op, op2 uint32) (eax, ebx, ecx, edx uint32) {
return 0, 0, 0, 0
}
xgetbv = func(index uint32) (eax, edx uint32) {
return 0, 0
}
rdtscpAsm = func() (eax, ebx, ecx, edx uint32) {
return 0, 0, 0, 0
}
}

3
vendor/github.com/klauspost/cpuid/generate.go generated vendored Normal file
View File

@@ -0,0 +1,3 @@
package cpuid
//go:generate go run private-gen.go

209
vendor/github.com/klauspost/cpuid/mockcpu_test.go generated vendored Normal file
View File

@@ -0,0 +1,209 @@
package cpuid
import (
"archive/zip"
"fmt"
"io/ioutil"
"sort"
"strings"
"testing"
)
type fakecpuid map[uint32][][]uint32
type idfuncs struct {
cpuid func(op uint32) (eax, ebx, ecx, edx uint32)
cpuidex func(op, op2 uint32) (eax, ebx, ecx, edx uint32)
xgetbv func(index uint32) (eax, edx uint32)
}
func (f fakecpuid) String() string {
var out = make([]string, 0, len(f))
for key, val := range f {
for _, v := range val {
out = append(out, fmt.Sprintf("CPUID %08x: [%08x, %08x, %08x, %08x]", key, v[0], v[1], v[2], v[3]))
}
}
sorter := sort.StringSlice(out)
sort.Sort(&sorter)
return strings.Join(sorter, "\n")
}
func mockCPU(def []byte) func() {
lines := strings.Split(string(def), "\n")
anyfound := false
fakeID := make(fakecpuid)
for _, line := range lines {
line = strings.Trim(line, "\r\t ")
if !strings.HasPrefix(line, "CPUID") {
continue
}
// Only collect for first cpu
if strings.HasPrefix(line, "CPUID 00000000") {
if anyfound {
break
}
}
if !strings.Contains(line, "-") {
//continue
}
items := strings.Split(line, ":")
if len(items) < 2 {
if len(line) == 51 || len(line) == 50 {
items = []string{line[0:14], line[15:]}
} else {
items = strings.Split(line, "\t")
if len(items) != 2 {
//fmt.Println("not found:", line, "len:", len(line))
continue
}
}
}
items = items[0:2]
vals := strings.Trim(items[1], "\r\n ")
var idV uint32
n, err := fmt.Sscanf(items[0], "CPUID %x", &idV)
if err != nil || n != 1 {
continue
}
existing, ok := fakeID[idV]
if !ok {
existing = make([][]uint32, 0)
}
values := make([]uint32, 4)
n, err = fmt.Sscanf(vals, "%x-%x-%x-%x", &values[0], &values[1], &values[2], &values[3])
if n != 4 || err != nil {
n, err = fmt.Sscanf(vals, "%x %x %x %x", &values[0], &values[1], &values[2], &values[3])
if n != 4 || err != nil {
//fmt.Println("scanned", vals, "got", n, "Err:", err)
continue
}
}
existing = append(existing, values)
fakeID[idV] = existing
anyfound = true
}
restorer := func(f idfuncs) func() {
return func() {
cpuid = f.cpuid
cpuidex = f.cpuidex
xgetbv = f.xgetbv
}
}(idfuncs{cpuid: cpuid, cpuidex: cpuidex, xgetbv: xgetbv})
cpuid = func(op uint32) (eax, ebx, ecx, edx uint32) {
if op == 0x80000000 || op == 0 {
var ok bool
_, ok = fakeID[op]
if !ok {
return 0, 0, 0, 0
}
}
first, ok := fakeID[op]
if !ok {
if op > maxFunctionID() {
panic(fmt.Sprintf("Base not found: %v, request:%#v\n", fakeID, op))
} else {
// we have some entries missing
return 0, 0, 0, 0
}
}
theid := first[0]
return theid[0], theid[1], theid[2], theid[3]
}
cpuidex = func(op, op2 uint32) (eax, ebx, ecx, edx uint32) {
if op == 0x80000000 {
var ok bool
_, ok = fakeID[op]
if !ok {
return 0, 0, 0, 0
}
}
first, ok := fakeID[op]
if !ok {
if op > maxExtendedFunction() {
panic(fmt.Sprintf("Extended not found Info: %v, request:%#v, %#v\n", fakeID, op, op2))
} else {
// we have some entries missing
return 0, 0, 0, 0
}
}
if int(op2) >= len(first) {
//fmt.Printf("Extended not found Info: %v, request:%#v, %#v\n", fakeID, op, op2)
return 0, 0, 0, 0
}
theid := first[op2]
return theid[0], theid[1], theid[2], theid[3]
}
xgetbv = func(index uint32) (eax, edx uint32) {
first, ok := fakeID[1]
if !ok {
panic(fmt.Sprintf("XGETBV not supported %v", fakeID))
}
second := first[0]
// ECX bit 26 must be set
if (second[2] & 1 << 26) == 0 {
panic(fmt.Sprintf("XGETBV not supported %v", fakeID))
}
// We don't have any data to return, unfortunately
return 0, 0
}
return restorer
}
func TestMocks(t *testing.T) {
zr, err := zip.OpenReader("testdata/cpuid_data.zip")
if err != nil {
t.Skip("No testdata:", err)
}
defer zr.Close()
for _, f := range zr.File {
rc, err := f.Open()
if err != nil {
t.Fatal(err)
}
content, err := ioutil.ReadAll(rc)
if err != nil {
t.Fatal(err)
}
rc.Close()
t.Log("Opening", f.FileInfo().Name())
restore := mockCPU(content)
Detect()
t.Log("Name:", CPU.BrandName)
n := maxFunctionID()
t.Logf("Max Function:0x%x\n", n)
n = maxExtendedFunction()
t.Logf("Max Extended Function:0x%x\n", n)
t.Log("PhysicalCores:", CPU.PhysicalCores)
t.Log("ThreadsPerCore:", CPU.ThreadsPerCore)
t.Log("LogicalCores:", CPU.LogicalCores)
t.Log("Family", CPU.Family, "Model:", CPU.Model)
t.Log("Features:", CPU.Features)
t.Log("Cacheline bytes:", CPU.CacheLine)
t.Log("L1 Instruction Cache:", CPU.Cache.L1I, "bytes")
t.Log("L1 Data Cache:", CPU.Cache.L1D, "bytes")
t.Log("L2 Cache:", CPU.Cache.L2, "bytes")
t.Log("L3 Cache:", CPU.Cache.L3, "bytes")
if CPU.LogicalCores > 0 && CPU.PhysicalCores > 0 {
if CPU.LogicalCores != CPU.PhysicalCores*CPU.ThreadsPerCore {
t.Fatalf("Core count mismatch, LogicalCores (%d) != PhysicalCores (%d) * CPU.ThreadsPerCore (%d)",
CPU.LogicalCores, CPU.PhysicalCores, CPU.ThreadsPerCore)
}
}
if CPU.ThreadsPerCore > 1 && !CPU.HTT() {
t.Fatalf("Hyperthreading not detected")
}
if CPU.ThreadsPerCore == 1 && CPU.HTT() {
t.Fatalf("Hyperthreading detected, but only 1 Thread per core")
}
restore()
}
Detect()
}

476
vendor/github.com/klauspost/cpuid/private-gen.go generated vendored Normal file
View File

@@ -0,0 +1,476 @@
// +build ignore
package main
import (
"bytes"
"fmt"
"go/ast"
"go/parser"
"go/printer"
"go/token"
"io"
"io/ioutil"
"log"
"os"
"reflect"
"strings"
"unicode"
"unicode/utf8"
)
var inFiles = []string{"cpuid.go", "cpuid_test.go"}
var copyFiles = []string{"cpuid_amd64.s", "cpuid_386.s", "detect_ref.go", "detect_intel.go"}
var fileSet = token.NewFileSet()
var reWrites = []rewrite{
initRewrite("CPUInfo -> cpuInfo"),
initRewrite("Vendor -> vendor"),
initRewrite("Flags -> flags"),
initRewrite("Detect -> detect"),
initRewrite("CPU -> cpu"),
}
var excludeNames = map[string]bool{"string": true, "join": true, "trim": true,
// cpuid_test.go
"t": true, "println": true, "logf": true, "log": true, "fatalf": true, "fatal": true,
}
var excludePrefixes = []string{"test", "benchmark"}
func main() {
Package := "private"
parserMode := parser.ParseComments
exported := make(map[string]rewrite)
for _, file := range inFiles {
in, err := os.Open(file)
if err != nil {
log.Fatalf("opening input", err)
}
src, err := ioutil.ReadAll(in)
if err != nil {
log.Fatalf("reading input", err)
}
astfile, err := parser.ParseFile(fileSet, file, src, parserMode)
if err != nil {
log.Fatalf("parsing input", err)
}
for _, rw := range reWrites {
astfile = rw(astfile)
}
// Inspect the AST and print all identifiers and literals.
var startDecl token.Pos
var endDecl token.Pos
ast.Inspect(astfile, func(n ast.Node) bool {
var s string
switch x := n.(type) {
case *ast.Ident:
if x.IsExported() {
t := strings.ToLower(x.Name)
for _, pre := range excludePrefixes {
if strings.HasPrefix(t, pre) {
return true
}
}
if excludeNames[t] != true {
//if x.Pos() > startDecl && x.Pos() < endDecl {
exported[x.Name] = initRewrite(x.Name + " -> " + t)
}
}
case *ast.GenDecl:
if x.Tok == token.CONST && x.Lparen > 0 {
startDecl = x.Lparen
endDecl = x.Rparen
// fmt.Printf("Decl:%s -> %s\n", fileSet.Position(startDecl), fileSet.Position(endDecl))
}
}
if s != "" {
fmt.Printf("%s:\t%s\n", fileSet.Position(n.Pos()), s)
}
return true
})
for _, rw := range exported {
astfile = rw(astfile)
}
var buf bytes.Buffer
printer.Fprint(&buf, fileSet, astfile)
// Remove package documentation and insert information
s := buf.String()
ind := strings.Index(buf.String(), "\npackage cpuid")
s = s[ind:]
s = "// Generated, DO NOT EDIT,\n" +
"// but copy it to your own project and rename the package.\n" +
"// See more at http://github.com/klauspost/cpuid\n" +
s
outputName := Package + string(os.PathSeparator) + file
err = ioutil.WriteFile(outputName, []byte(s), 0644)
if err != nil {
log.Fatalf("writing output: %s", err)
}
log.Println("Generated", outputName)
}
for _, file := range copyFiles {
dst := ""
if strings.HasPrefix(file, "cpuid") {
dst = Package + string(os.PathSeparator) + file
} else {
dst = Package + string(os.PathSeparator) + "cpuid_" + file
}
err := copyFile(file, dst)
if err != nil {
log.Fatalf("copying file: %s", err)
}
log.Println("Copied", dst)
}
}
// CopyFile copies a file from src to dst. If src and dst files exist, and are
// the same, then return success. Copy the file contents from src to dst.
func copyFile(src, dst string) (err error) {
sfi, err := os.Stat(src)
if err != nil {
return
}
if !sfi.Mode().IsRegular() {
// cannot copy non-regular files (e.g., directories,
// symlinks, devices, etc.)
return fmt.Errorf("CopyFile: non-regular source file %s (%q)", sfi.Name(), sfi.Mode().String())
}
dfi, err := os.Stat(dst)
if err != nil {
if !os.IsNotExist(err) {
return
}
} else {
if !(dfi.Mode().IsRegular()) {
return fmt.Errorf("CopyFile: non-regular destination file %s (%q)", dfi.Name(), dfi.Mode().String())
}
if os.SameFile(sfi, dfi) {
return
}
}
err = copyFileContents(src, dst)
return
}
// copyFileContents copies the contents of the file named src to the file named
// by dst. The file will be created if it does not already exist. If the
// destination file exists, all it's contents will be replaced by the contents
// of the source file.
func copyFileContents(src, dst string) (err error) {
in, err := os.Open(src)
if err != nil {
return
}
defer in.Close()
out, err := os.Create(dst)
if err != nil {
return
}
defer func() {
cerr := out.Close()
if err == nil {
err = cerr
}
}()
if _, err = io.Copy(out, in); err != nil {
return
}
err = out.Sync()
return
}
type rewrite func(*ast.File) *ast.File
// Mostly copied from gofmt
func initRewrite(rewriteRule string) rewrite {
f := strings.Split(rewriteRule, "->")
if len(f) != 2 {
fmt.Fprintf(os.Stderr, "rewrite rule must be of the form 'pattern -> replacement'\n")
os.Exit(2)
}
pattern := parseExpr(f[0], "pattern")
replace := parseExpr(f[1], "replacement")
return func(p *ast.File) *ast.File { return rewriteFile(pattern, replace, p) }
}
// parseExpr parses s as an expression.
// It might make sense to expand this to allow statement patterns,
// but there are problems with preserving formatting and also
// with what a wildcard for a statement looks like.
func parseExpr(s, what string) ast.Expr {
x, err := parser.ParseExpr(s)
if err != nil {
fmt.Fprintf(os.Stderr, "parsing %s %s at %s\n", what, s, err)
os.Exit(2)
}
return x
}
// Keep this function for debugging.
/*
func dump(msg string, val reflect.Value) {
fmt.Printf("%s:\n", msg)
ast.Print(fileSet, val.Interface())
fmt.Println()
}
*/
// rewriteFile applies the rewrite rule 'pattern -> replace' to an entire file.
func rewriteFile(pattern, replace ast.Expr, p *ast.File) *ast.File {
cmap := ast.NewCommentMap(fileSet, p, p.Comments)
m := make(map[string]reflect.Value)
pat := reflect.ValueOf(pattern)
repl := reflect.ValueOf(replace)
var rewriteVal func(val reflect.Value) reflect.Value
rewriteVal = func(val reflect.Value) reflect.Value {
// don't bother if val is invalid to start with
if !val.IsValid() {
return reflect.Value{}
}
for k := range m {
delete(m, k)
}
val = apply(rewriteVal, val)
if match(m, pat, val) {
val = subst(m, repl, reflect.ValueOf(val.Interface().(ast.Node).Pos()))
}
return val
}
r := apply(rewriteVal, reflect.ValueOf(p)).Interface().(*ast.File)
r.Comments = cmap.Filter(r).Comments() // recreate comments list
return r
}
// set is a wrapper for x.Set(y); it protects the caller from panics if x cannot be changed to y.
func set(x, y reflect.Value) {
// don't bother if x cannot be set or y is invalid
if !x.CanSet() || !y.IsValid() {
return
}
defer func() {
if x := recover(); x != nil {
if s, ok := x.(string); ok &&
(strings.Contains(s, "type mismatch") || strings.Contains(s, "not assignable")) {
// x cannot be set to y - ignore this rewrite
return
}
panic(x)
}
}()
x.Set(y)
}
// Values/types for special cases.
var (
objectPtrNil = reflect.ValueOf((*ast.Object)(nil))
scopePtrNil = reflect.ValueOf((*ast.Scope)(nil))
identType = reflect.TypeOf((*ast.Ident)(nil))
objectPtrType = reflect.TypeOf((*ast.Object)(nil))
positionType = reflect.TypeOf(token.NoPos)
callExprType = reflect.TypeOf((*ast.CallExpr)(nil))
scopePtrType = reflect.TypeOf((*ast.Scope)(nil))
)
// apply replaces each AST field x in val with f(x), returning val.
// To avoid extra conversions, f operates on the reflect.Value form.
func apply(f func(reflect.Value) reflect.Value, val reflect.Value) reflect.Value {
if !val.IsValid() {
return reflect.Value{}
}
// *ast.Objects introduce cycles and are likely incorrect after
// rewrite; don't follow them but replace with nil instead
if val.Type() == objectPtrType {
return objectPtrNil
}
// similarly for scopes: they are likely incorrect after a rewrite;
// replace them with nil
if val.Type() == scopePtrType {
return scopePtrNil
}
switch v := reflect.Indirect(val); v.Kind() {
case reflect.Slice:
for i := 0; i < v.Len(); i++ {
e := v.Index(i)
set(e, f(e))
}
case reflect.Struct:
for i := 0; i < v.NumField(); i++ {
e := v.Field(i)
set(e, f(e))
}
case reflect.Interface:
e := v.Elem()
set(v, f(e))
}
return val
}
func isWildcard(s string) bool {
rune, size := utf8.DecodeRuneInString(s)
return size == len(s) && unicode.IsLower(rune)
}
// match returns true if pattern matches val,
// recording wildcard submatches in m.
// If m == nil, match checks whether pattern == val.
func match(m map[string]reflect.Value, pattern, val reflect.Value) bool {
// Wildcard matches any expression. If it appears multiple
// times in the pattern, it must match the same expression
// each time.
if m != nil && pattern.IsValid() && pattern.Type() == identType {
name := pattern.Interface().(*ast.Ident).Name
if isWildcard(name) && val.IsValid() {
// wildcards only match valid (non-nil) expressions.
if _, ok := val.Interface().(ast.Expr); ok && !val.IsNil() {
if old, ok := m[name]; ok {
return match(nil, old, val)
}
m[name] = val
return true
}
}
}
// Otherwise, pattern and val must match recursively.
if !pattern.IsValid() || !val.IsValid() {
return !pattern.IsValid() && !val.IsValid()
}
if pattern.Type() != val.Type() {
return false
}
// Special cases.
switch pattern.Type() {
case identType:
// For identifiers, only the names need to match
// (and none of the other *ast.Object information).
// This is a common case, handle it all here instead
// of recursing down any further via reflection.
p := pattern.Interface().(*ast.Ident)
v := val.Interface().(*ast.Ident)
return p == nil && v == nil || p != nil && v != nil && p.Name == v.Name
case objectPtrType, positionType:
// object pointers and token positions always match
return true
case callExprType:
// For calls, the Ellipsis fields (token.Position) must
// match since that is how f(x) and f(x...) are different.
// Check them here but fall through for the remaining fields.
p := pattern.Interface().(*ast.CallExpr)
v := val.Interface().(*ast.CallExpr)
if p.Ellipsis.IsValid() != v.Ellipsis.IsValid() {
return false
}
}
p := reflect.Indirect(pattern)
v := reflect.Indirect(val)
if !p.IsValid() || !v.IsValid() {
return !p.IsValid() && !v.IsValid()
}
switch p.Kind() {
case reflect.Slice:
if p.Len() != v.Len() {
return false
}
for i := 0; i < p.Len(); i++ {
if !match(m, p.Index(i), v.Index(i)) {
return false
}
}
return true
case reflect.Struct:
for i := 0; i < p.NumField(); i++ {
if !match(m, p.Field(i), v.Field(i)) {
return false
}
}
return true
case reflect.Interface:
return match(m, p.Elem(), v.Elem())
}
// Handle token integers, etc.
return p.Interface() == v.Interface()
}
// subst returns a copy of pattern with values from m substituted in place
// of wildcards and pos used as the position of tokens from the pattern.
// if m == nil, subst returns a copy of pattern and doesn't change the line
// number information.
func subst(m map[string]reflect.Value, pattern reflect.Value, pos reflect.Value) reflect.Value {
if !pattern.IsValid() {
return reflect.Value{}
}
// Wildcard gets replaced with map value.
if m != nil && pattern.Type() == identType {
name := pattern.Interface().(*ast.Ident).Name
if isWildcard(name) {
if old, ok := m[name]; ok {
return subst(nil, old, reflect.Value{})
}
}
}
if pos.IsValid() && pattern.Type() == positionType {
// use new position only if old position was valid in the first place
if old := pattern.Interface().(token.Pos); !old.IsValid() {
return pattern
}
return pos
}
// Otherwise copy.
switch p := pattern; p.Kind() {
case reflect.Slice:
v := reflect.MakeSlice(p.Type(), p.Len(), p.Len())
for i := 0; i < p.Len(); i++ {
v.Index(i).Set(subst(m, p.Index(i), pos))
}
return v
case reflect.Struct:
v := reflect.New(p.Type()).Elem()
for i := 0; i < p.NumField(); i++ {
v.Field(i).Set(subst(m, p.Field(i), pos))
}
return v
case reflect.Ptr:
v := reflect.New(p.Type()).Elem()
if elem := p.Elem(); elem.IsValid() {
v.Set(subst(m, elem, pos).Addr())
}
return v
case reflect.Interface:
v := reflect.New(p.Type()).Elem()
if elem := p.Elem(); elem.IsValid() {
v.Set(subst(m, elem, pos))
}
return v
}
return pattern
}

6
vendor/github.com/klauspost/cpuid/private/README.md generated vendored Normal file
View File

@@ -0,0 +1,6 @@
# cpuid private
This is a specially converted of the cpuid package, so it can be included in
a package without exporting anything.
Package home: https://github.com/klauspost/cpuid

987
vendor/github.com/klauspost/cpuid/private/cpuid.go generated vendored Normal file
View File

@@ -0,0 +1,987 @@
// Generated, DO NOT EDIT,
// but copy it to your own project and rename the package.
// See more at http://github.com/klauspost/cpuid
package cpuid
import (
"strings"
)
// Vendor is a representation of a CPU vendor.
type vendor int
const (
other vendor = iota
intel
amd
via
transmeta
nsc
kvm // Kernel-based Virtual Machine
msvm // Microsoft Hyper-V or Windows Virtual PC
vmware
xenhvm
)
const (
cmov = 1 << iota // i686 CMOV
nx // NX (No-Execute) bit
amd3dnow // AMD 3DNOW
amd3dnowext // AMD 3DNowExt
mmx // standard MMX
mmxext // SSE integer functions or AMD MMX ext
sse // SSE functions
sse2 // P4 SSE functions
sse3 // Prescott SSE3 functions
ssse3 // Conroe SSSE3 functions
sse4 // Penryn SSE4.1 functions
sse4a // AMD Barcelona microarchitecture SSE4a instructions
sse42 // Nehalem SSE4.2 functions
avx // AVX functions
avx2 // AVX2 functions
fma3 // Intel FMA 3
fma4 // Bulldozer FMA4 functions
xop // Bulldozer XOP functions
f16c // Half-precision floating-point conversion
bmi1 // Bit Manipulation Instruction Set 1
bmi2 // Bit Manipulation Instruction Set 2
tbm // AMD Trailing Bit Manipulation
lzcnt // LZCNT instruction
popcnt // POPCNT instruction
aesni // Advanced Encryption Standard New Instructions
clmul // Carry-less Multiplication
htt // Hyperthreading (enabled)
hle // Hardware Lock Elision
rtm // Restricted Transactional Memory
rdrand // RDRAND instruction is available
rdseed // RDSEED instruction is available
adx // Intel ADX (Multi-Precision Add-Carry Instruction Extensions)
sha // Intel SHA Extensions
avx512f // AVX-512 Foundation
avx512dq // AVX-512 Doubleword and Quadword Instructions
avx512ifma // AVX-512 Integer Fused Multiply-Add Instructions
avx512pf // AVX-512 Prefetch Instructions
avx512er // AVX-512 Exponential and Reciprocal Instructions
avx512cd // AVX-512 Conflict Detection Instructions
avx512bw // AVX-512 Byte and Word Instructions
avx512vl // AVX-512 Vector Length Extensions
avx512vbmi // AVX-512 Vector Bit Manipulation Instructions
mpx // Intel MPX (Memory Protection Extensions)
erms // Enhanced REP MOVSB/STOSB
rdtscp // RDTSCP Instruction
cx16 // CMPXCHG16B Instruction
// Performance indicators
sse2slow // SSE2 is supported, but usually not faster
sse3slow // SSE3 is supported, but usually not faster
atom // Atom processor, some SSSE3 instructions are slower
)
var flagNames = map[flags]string{
cmov: "CMOV", // i686 CMOV
nx: "NX", // NX (No-Execute) bit
amd3dnow: "AMD3DNOW", // AMD 3DNOW
amd3dnowext: "AMD3DNOWEXT", // AMD 3DNowExt
mmx: "MMX", // Standard MMX
mmxext: "MMXEXT", // SSE integer functions or AMD MMX ext
sse: "SSE", // SSE functions
sse2: "SSE2", // P4 SSE2 functions
sse3: "SSE3", // Prescott SSE3 functions
ssse3: "SSSE3", // Conroe SSSE3 functions
sse4: "SSE4.1", // Penryn SSE4.1 functions
sse4a: "SSE4A", // AMD Barcelona microarchitecture SSE4a instructions
sse42: "SSE4.2", // Nehalem SSE4.2 functions
avx: "AVX", // AVX functions
avx2: "AVX2", // AVX functions
fma3: "FMA3", // Intel FMA 3
fma4: "FMA4", // Bulldozer FMA4 functions
xop: "XOP", // Bulldozer XOP functions
f16c: "F16C", // Half-precision floating-point conversion
bmi1: "BMI1", // Bit Manipulation Instruction Set 1
bmi2: "BMI2", // Bit Manipulation Instruction Set 2
tbm: "TBM", // AMD Trailing Bit Manipulation
lzcnt: "LZCNT", // LZCNT instruction
popcnt: "POPCNT", // POPCNT instruction
aesni: "AESNI", // Advanced Encryption Standard New Instructions
clmul: "CLMUL", // Carry-less Multiplication
htt: "HTT", // Hyperthreading (enabled)
hle: "HLE", // Hardware Lock Elision
rtm: "RTM", // Restricted Transactional Memory
rdrand: "RDRAND", // RDRAND instruction is available
rdseed: "RDSEED", // RDSEED instruction is available
adx: "ADX", // Intel ADX (Multi-Precision Add-Carry Instruction Extensions)
sha: "SHA", // Intel SHA Extensions
avx512f: "AVX512F", // AVX-512 Foundation
avx512dq: "AVX512DQ", // AVX-512 Doubleword and Quadword Instructions
avx512ifma: "AVX512IFMA", // AVX-512 Integer Fused Multiply-Add Instructions
avx512pf: "AVX512PF", // AVX-512 Prefetch Instructions
avx512er: "AVX512ER", // AVX-512 Exponential and Reciprocal Instructions
avx512cd: "AVX512CD", // AVX-512 Conflict Detection Instructions
avx512bw: "AVX512BW", // AVX-512 Byte and Word Instructions
avx512vl: "AVX512VL", // AVX-512 Vector Length Extensions
avx512vbmi: "AVX512VBMI", // AVX-512 Vector Bit Manipulation Instructions
mpx: "MPX", // Intel MPX (Memory Protection Extensions)
erms: "ERMS", // Enhanced REP MOVSB/STOSB
rdtscp: "RDTSCP", // RDTSCP Instruction
cx16: "CX16", // CMPXCHG16B Instruction
// Performance indicators
sse2slow: "SSE2SLOW", // SSE2 supported, but usually not faster
sse3slow: "SSE3SLOW", // SSE3 supported, but usually not faster
atom: "ATOM", // Atom processor, some SSSE3 instructions are slower
}
// CPUInfo contains information about the detected system CPU.
type cpuInfo struct {
brandname string // Brand name reported by the CPU
vendorid vendor // Comparable CPU vendor ID
features flags // Features of the CPU
physicalcores int // Number of physical processor cores in your CPU. Will be 0 if undetectable.
threadspercore int // Number of threads per physical core. Will be 1 if undetectable.
logicalcores int // Number of physical cores times threads that can run on each core through the use of hyperthreading. Will be 0 if undetectable.
family int // CPU family number
model int // CPU model number
cacheline int // Cache line size in bytes. Will be 0 if undetectable.
cache struct {
l1i int // L1 Instruction Cache (per core or shared). Will be -1 if undetected
l1d int // L1 Data Cache (per core or shared). Will be -1 if undetected
l2 int // L2 Cache (per core or shared). Will be -1 if undetected
l3 int // L3 Instruction Cache (per core or shared). Will be -1 if undetected
}
maxFunc uint32
maxExFunc uint32
}
var cpuid func(op uint32) (eax, ebx, ecx, edx uint32)
var cpuidex func(op, op2 uint32) (eax, ebx, ecx, edx uint32)
var xgetbv func(index uint32) (eax, edx uint32)
var rdtscpAsm func() (eax, ebx, ecx, edx uint32)
// CPU contains information about the CPU as detected on startup,
// or when Detect last was called.
//
// Use this as the primary entry point to you data,
// this way queries are
var cpu cpuInfo
func init() {
initCPU()
detect()
}
// Detect will re-detect current CPU info.
// This will replace the content of the exported CPU variable.
//
// Unless you expect the CPU to change while you are running your program
// you should not need to call this function.
// If you call this, you must ensure that no other goroutine is accessing the
// exported CPU variable.
func detect() {
cpu.maxFunc = maxFunctionID()
cpu.maxExFunc = maxExtendedFunction()
cpu.brandname = brandName()
cpu.cacheline = cacheLine()
cpu.family, cpu.model = familyModel()
cpu.features = support()
cpu.threadspercore = threadsPerCore()
cpu.logicalcores = logicalCores()
cpu.physicalcores = physicalCores()
cpu.vendorid = vendorID()
cpu.cacheSize()
}
// Generated here: http://play.golang.org/p/BxFH2Gdc0G
// Cmov indicates support of CMOV instructions
func (c cpuInfo) cmov() bool {
return c.features&cmov != 0
}
// Amd3dnow indicates support of AMD 3DNOW! instructions
func (c cpuInfo) amd3dnow() bool {
return c.features&amd3dnow != 0
}
// Amd3dnowExt indicates support of AMD 3DNOW! Extended instructions
func (c cpuInfo) amd3dnowext() bool {
return c.features&amd3dnowext != 0
}
// MMX indicates support of MMX instructions
func (c cpuInfo) mmx() bool {
return c.features&mmx != 0
}
// MMXExt indicates support of MMXEXT instructions
// (SSE integer functions or AMD MMX ext)
func (c cpuInfo) mmxext() bool {
return c.features&mmxext != 0
}
// SSE indicates support of SSE instructions
func (c cpuInfo) sse() bool {
return c.features&sse != 0
}
// SSE2 indicates support of SSE 2 instructions
func (c cpuInfo) sse2() bool {
return c.features&sse2 != 0
}
// SSE3 indicates support of SSE 3 instructions
func (c cpuInfo) sse3() bool {
return c.features&sse3 != 0
}
// SSSE3 indicates support of SSSE 3 instructions
func (c cpuInfo) ssse3() bool {
return c.features&ssse3 != 0
}
// SSE4 indicates support of SSE 4 (also called SSE 4.1) instructions
func (c cpuInfo) sse4() bool {
return c.features&sse4 != 0
}
// SSE42 indicates support of SSE4.2 instructions
func (c cpuInfo) sse42() bool {
return c.features&sse42 != 0
}
// AVX indicates support of AVX instructions
// and operating system support of AVX instructions
func (c cpuInfo) avx() bool {
return c.features&avx != 0
}
// AVX2 indicates support of AVX2 instructions
func (c cpuInfo) avx2() bool {
return c.features&avx2 != 0
}
// FMA3 indicates support of FMA3 instructions
func (c cpuInfo) fma3() bool {
return c.features&fma3 != 0
}
// FMA4 indicates support of FMA4 instructions
func (c cpuInfo) fma4() bool {
return c.features&fma4 != 0
}
// XOP indicates support of XOP instructions
func (c cpuInfo) xop() bool {
return c.features&xop != 0
}
// F16C indicates support of F16C instructions
func (c cpuInfo) f16c() bool {
return c.features&f16c != 0
}
// BMI1 indicates support of BMI1 instructions
func (c cpuInfo) bmi1() bool {
return c.features&bmi1 != 0
}
// BMI2 indicates support of BMI2 instructions
func (c cpuInfo) bmi2() bool {
return c.features&bmi2 != 0
}
// TBM indicates support of TBM instructions
// (AMD Trailing Bit Manipulation)
func (c cpuInfo) tbm() bool {
return c.features&tbm != 0
}
// Lzcnt indicates support of LZCNT instruction
func (c cpuInfo) lzcnt() bool {
return c.features&lzcnt != 0
}
// Popcnt indicates support of POPCNT instruction
func (c cpuInfo) popcnt() bool {
return c.features&popcnt != 0
}
// HTT indicates the processor has Hyperthreading enabled
func (c cpuInfo) htt() bool {
return c.features&htt != 0
}
// SSE2Slow indicates that SSE2 may be slow on this processor
func (c cpuInfo) sse2slow() bool {
return c.features&sse2slow != 0
}
// SSE3Slow indicates that SSE3 may be slow on this processor
func (c cpuInfo) sse3slow() bool {
return c.features&sse3slow != 0
}
// AesNi indicates support of AES-NI instructions
// (Advanced Encryption Standard New Instructions)
func (c cpuInfo) aesni() bool {
return c.features&aesni != 0
}
// Clmul indicates support of CLMUL instructions
// (Carry-less Multiplication)
func (c cpuInfo) clmul() bool {
return c.features&clmul != 0
}
// NX indicates support of NX (No-Execute) bit
func (c cpuInfo) nx() bool {
return c.features&nx != 0
}
// SSE4A indicates support of AMD Barcelona microarchitecture SSE4a instructions
func (c cpuInfo) sse4a() bool {
return c.features&sse4a != 0
}
// HLE indicates support of Hardware Lock Elision
func (c cpuInfo) hle() bool {
return c.features&hle != 0
}
// RTM indicates support of Restricted Transactional Memory
func (c cpuInfo) rtm() bool {
return c.features&rtm != 0
}
// Rdrand indicates support of RDRAND instruction is available
func (c cpuInfo) rdrand() bool {
return c.features&rdrand != 0
}
// Rdseed indicates support of RDSEED instruction is available
func (c cpuInfo) rdseed() bool {
return c.features&rdseed != 0
}
// ADX indicates support of Intel ADX (Multi-Precision Add-Carry Instruction Extensions)
func (c cpuInfo) adx() bool {
return c.features&adx != 0
}
// SHA indicates support of Intel SHA Extensions
func (c cpuInfo) sha() bool {
return c.features&sha != 0
}
// AVX512F indicates support of AVX-512 Foundation
func (c cpuInfo) avx512f() bool {
return c.features&avx512f != 0
}
// AVX512DQ indicates support of AVX-512 Doubleword and Quadword Instructions
func (c cpuInfo) avx512dq() bool {
return c.features&avx512dq != 0
}
// AVX512IFMA indicates support of AVX-512 Integer Fused Multiply-Add Instructions
func (c cpuInfo) avx512ifma() bool {
return c.features&avx512ifma != 0
}
// AVX512PF indicates support of AVX-512 Prefetch Instructions
func (c cpuInfo) avx512pf() bool {
return c.features&avx512pf != 0
}
// AVX512ER indicates support of AVX-512 Exponential and Reciprocal Instructions
func (c cpuInfo) avx512er() bool {
return c.features&avx512er != 0
}
// AVX512CD indicates support of AVX-512 Conflict Detection Instructions
func (c cpuInfo) avx512cd() bool {
return c.features&avx512cd != 0
}
// AVX512BW indicates support of AVX-512 Byte and Word Instructions
func (c cpuInfo) avx512bw() bool {
return c.features&avx512bw != 0
}
// AVX512VL indicates support of AVX-512 Vector Length Extensions
func (c cpuInfo) avx512vl() bool {
return c.features&avx512vl != 0
}
// AVX512VBMI indicates support of AVX-512 Vector Bit Manipulation Instructions
func (c cpuInfo) avx512vbmi() bool {
return c.features&avx512vbmi != 0
}
// MPX indicates support of Intel MPX (Memory Protection Extensions)
func (c cpuInfo) mpx() bool {
return c.features&mpx != 0
}
// ERMS indicates support of Enhanced REP MOVSB/STOSB
func (c cpuInfo) erms() bool {
return c.features&erms != 0
}
func (c cpuInfo) rdtscp() bool {
return c.features&rdtscp != 0
}
func (c cpuInfo) cx16() bool {
return c.features&cx16 != 0
}
// Atom indicates an Atom processor
func (c cpuInfo) atom() bool {
return c.features&atom != 0
}
// Intel returns true if vendor is recognized as Intel
func (c cpuInfo) intel() bool {
return c.vendorid == intel
}
// AMD returns true if vendor is recognized as AMD
func (c cpuInfo) amd() bool {
return c.vendorid == amd
}
// Transmeta returns true if vendor is recognized as Transmeta
func (c cpuInfo) transmeta() bool {
return c.vendorid == transmeta
}
// NSC returns true if vendor is recognized as National Semiconductor
func (c cpuInfo) nsc() bool {
return c.vendorid == nsc
}
// VIA returns true if vendor is recognized as VIA
func (c cpuInfo) via() bool {
return c.vendorid == via
}
// RTCounter returns the 64-bit time-stamp counter
// Uses the RDTSCP instruction. The value 0 is returned
// if the CPU does not support the instruction.
func (c cpuInfo) rtcounter() uint64 {
if !c.rdtscp() {
return 0
}
a, _, _, d := rdtscpAsm()
return uint64(a) | (uint64(d) << 32)
}
// Ia32TscAux returns the IA32_TSC_AUX part of the RDTSCP.
// This variable is OS dependent, but on Linux contains information
// about the current cpu/core the code is running on.
// If the RDTSCP instruction isn't supported on the CPU, the value 0 is returned.
func (c cpuInfo) ia32tscaux() uint32 {
if !c.rdtscp() {
return 0
}
_, _, ecx, _ := rdtscpAsm()
return ecx
}
// LogicalCPU will return the Logical CPU the code is currently executing on.
// This is likely to change when the OS re-schedules the running thread
// to another CPU.
// If the current core cannot be detected, -1 will be returned.
func (c cpuInfo) logicalcpu() int {
if c.maxFunc < 1 {
return -1
}
_, ebx, _, _ := cpuid(1)
return int(ebx >> 24)
}
// VM Will return true if the cpu id indicates we are in
// a virtual machine. This is only a hint, and will very likely
// have many false negatives.
func (c cpuInfo) vm() bool {
switch c.vendorid {
case msvm, kvm, vmware, xenhvm:
return true
}
return false
}
// Flags contains detected cpu features and caracteristics
type flags uint64
// String returns a string representation of the detected
// CPU features.
func (f flags) String() string {
return strings.Join(f.strings(), ",")
}
// Strings returns and array of the detected features.
func (f flags) strings() []string {
s := support()
r := make([]string, 0, 20)
for i := uint(0); i < 64; i++ {
key := flags(1 << i)
val := flagNames[key]
if s&key != 0 {
r = append(r, val)
}
}
return r
}
func maxExtendedFunction() uint32 {
eax, _, _, _ := cpuid(0x80000000)
return eax
}
func maxFunctionID() uint32 {
a, _, _, _ := cpuid(0)
return a
}
func brandName() string {
if maxExtendedFunction() >= 0x80000004 {
v := make([]uint32, 0, 48)
for i := uint32(0); i < 3; i++ {
a, b, c, d := cpuid(0x80000002 + i)
v = append(v, a, b, c, d)
}
return strings.Trim(string(valAsString(v...)), " ")
}
return "unknown"
}
func threadsPerCore() int {
mfi := maxFunctionID()
if mfi < 0x4 || vendorID() != intel {
return 1
}
if mfi < 0xb {
_, b, _, d := cpuid(1)
if (d & (1 << 28)) != 0 {
// v will contain logical core count
v := (b >> 16) & 255
if v > 1 {
a4, _, _, _ := cpuid(4)
// physical cores
v2 := (a4 >> 26) + 1
if v2 > 0 {
return int(v) / int(v2)
}
}
}
return 1
}
_, b, _, _ := cpuidex(0xb, 0)
if b&0xffff == 0 {
return 1
}
return int(b & 0xffff)
}
func logicalCores() int {
mfi := maxFunctionID()
switch vendorID() {
case intel:
// Use this on old Intel processors
if mfi < 0xb {
if mfi < 1 {
return 0
}
// CPUID.1:EBX[23:16] represents the maximum number of addressable IDs (initial APIC ID)
// that can be assigned to logical processors in a physical package.
// The value may not be the same as the number of logical processors that are present in the hardware of a physical package.
_, ebx, _, _ := cpuid(1)
logical := (ebx >> 16) & 0xff
return int(logical)
}
_, b, _, _ := cpuidex(0xb, 1)
return int(b & 0xffff)
case amd:
_, b, _, _ := cpuid(1)
return int((b >> 16) & 0xff)
default:
return 0
}
}
func familyModel() (int, int) {
if maxFunctionID() < 0x1 {
return 0, 0
}
eax, _, _, _ := cpuid(1)
family := ((eax >> 8) & 0xf) + ((eax >> 20) & 0xff)
model := ((eax >> 4) & 0xf) + ((eax >> 12) & 0xf0)
return int(family), int(model)
}
func physicalCores() int {
switch vendorID() {
case intel:
return logicalCores() / threadsPerCore()
case amd:
if maxExtendedFunction() >= 0x80000008 {
_, _, c, _ := cpuid(0x80000008)
return int(c&0xff) + 1
}
}
return 0
}
// Except from http://en.wikipedia.org/wiki/CPUID#EAX.3D0:_Get_vendor_ID
var vendorMapping = map[string]vendor{
"AMDisbetter!": amd,
"AuthenticAMD": amd,
"CentaurHauls": via,
"GenuineIntel": intel,
"TransmetaCPU": transmeta,
"GenuineTMx86": transmeta,
"Geode by NSC": nsc,
"VIA VIA VIA ": via,
"KVMKVMKVMKVM": kvm,
"Microsoft Hv": msvm,
"VMwareVMware": vmware,
"XenVMMXenVMM": xenhvm,
}
func vendorID() vendor {
_, b, c, d := cpuid(0)
v := valAsString(b, d, c)
vend, ok := vendorMapping[string(v)]
if !ok {
return other
}
return vend
}
func cacheLine() int {
if maxFunctionID() < 0x1 {
return 0
}
_, ebx, _, _ := cpuid(1)
cache := (ebx & 0xff00) >> 5 // cflush size
if cache == 0 && maxExtendedFunction() >= 0x80000006 {
_, _, ecx, _ := cpuid(0x80000006)
cache = ecx & 0xff // cacheline size
}
// TODO: Read from Cache and TLB Information
return int(cache)
}
func (c *cpuInfo) cacheSize() {
c.cache.l1d = -1
c.cache.l1i = -1
c.cache.l2 = -1
c.cache.l3 = -1
vendor := vendorID()
switch vendor {
case intel:
if maxFunctionID() < 4 {
return
}
for i := uint32(0); ; i++ {
eax, ebx, ecx, _ := cpuidex(4, i)
cacheType := eax & 15
if cacheType == 0 {
break
}
cacheLevel := (eax >> 5) & 7
coherency := int(ebx&0xfff) + 1
partitions := int((ebx>>12)&0x3ff) + 1
associativity := int((ebx>>22)&0x3ff) + 1
sets := int(ecx) + 1
size := associativity * partitions * coherency * sets
switch cacheLevel {
case 1:
if cacheType == 1 {
// 1 = Data Cache
c.cache.l1d = size
} else if cacheType == 2 {
// 2 = Instruction Cache
c.cache.l1i = size
} else {
if c.cache.l1d < 0 {
c.cache.l1i = size
}
if c.cache.l1i < 0 {
c.cache.l1i = size
}
}
case 2:
c.cache.l2 = size
case 3:
c.cache.l3 = size
}
}
case amd:
// Untested.
if maxExtendedFunction() < 0x80000005 {
return
}
_, _, ecx, edx := cpuid(0x80000005)
c.cache.l1d = int(((ecx >> 24) & 0xFF) * 1024)
c.cache.l1i = int(((edx >> 24) & 0xFF) * 1024)
if maxExtendedFunction() < 0x80000006 {
return
}
_, _, ecx, _ = cpuid(0x80000006)
c.cache.l2 = int(((ecx >> 16) & 0xFFFF) * 1024)
}
return
}
func support() flags {
mfi := maxFunctionID()
vend := vendorID()
if mfi < 0x1 {
return 0
}
rval := uint64(0)
_, _, c, d := cpuid(1)
if (d & (1 << 15)) != 0 {
rval |= cmov
}
if (d & (1 << 23)) != 0 {
rval |= mmx
}
if (d & (1 << 25)) != 0 {
rval |= mmxext
}
if (d & (1 << 25)) != 0 {
rval |= sse
}
if (d & (1 << 26)) != 0 {
rval |= sse2
}
if (c & 1) != 0 {
rval |= sse3
}
if (c & 0x00000200) != 0 {
rval |= ssse3
}
if (c & 0x00080000) != 0 {
rval |= sse4
}
if (c & 0x00100000) != 0 {
rval |= sse42
}
if (c & (1 << 25)) != 0 {
rval |= aesni
}
if (c & (1 << 1)) != 0 {
rval |= clmul
}
if c&(1<<23) != 0 {
rval |= popcnt
}
if c&(1<<30) != 0 {
rval |= rdrand
}
if c&(1<<29) != 0 {
rval |= f16c
}
if c&(1<<13) != 0 {
rval |= cx16
}
if vend == intel && (d&(1<<28)) != 0 && mfi >= 4 {
if threadsPerCore() > 1 {
rval |= htt
}
}
// Check XGETBV, OXSAVE and AVX bits
if c&(1<<26) != 0 && c&(1<<27) != 0 && c&(1<<28) != 0 {
// Check for OS support
eax, _ := xgetbv(0)
if (eax & 0x6) == 0x6 {
rval |= avx
if (c & 0x00001000) != 0 {
rval |= fma3
}
}
}
// Check AVX2, AVX2 requires OS support, but BMI1/2 don't.
if mfi >= 7 {
_, ebx, ecx, _ := cpuidex(7, 0)
if (rval&avx) != 0 && (ebx&0x00000020) != 0 {
rval |= avx2
}
if (ebx & 0x00000008) != 0 {
rval |= bmi1
if (ebx & 0x00000100) != 0 {
rval |= bmi2
}
}
if ebx&(1<<4) != 0 {
rval |= hle
}
if ebx&(1<<9) != 0 {
rval |= erms
}
if ebx&(1<<11) != 0 {
rval |= rtm
}
if ebx&(1<<14) != 0 {
rval |= mpx
}
if ebx&(1<<18) != 0 {
rval |= rdseed
}
if ebx&(1<<19) != 0 {
rval |= adx
}
if ebx&(1<<29) != 0 {
rval |= sha
}
// Only detect AVX-512 features if XGETBV is supported
if c&((1<<26)|(1<<27)) == (1<<26)|(1<<27) {
// Check for OS support
eax, _ := xgetbv(0)
// Verify that XCR0[7:5] = ‘111b’ (OPMASK state, upper 256-bit of ZMM0-ZMM15 and
// ZMM16-ZMM31 state are enabled by OS)
/// and that XCR0[2:1] = ‘11b’ (XMM state and YMM state are enabled by OS).
if (eax>>5)&7 == 7 && (eax>>1)&3 == 3 {
if ebx&(1<<16) != 0 {
rval |= avx512f
}
if ebx&(1<<17) != 0 {
rval |= avx512dq
}
if ebx&(1<<21) != 0 {
rval |= avx512ifma
}
if ebx&(1<<26) != 0 {
rval |= avx512pf
}
if ebx&(1<<27) != 0 {
rval |= avx512er
}
if ebx&(1<<28) != 0 {
rval |= avx512cd
}
if ebx&(1<<30) != 0 {
rval |= avx512bw
}
if ebx&(1<<31) != 0 {
rval |= avx512vl
}
// ecx
if ecx&(1<<1) != 0 {
rval |= avx512vbmi
}
}
}
}
if maxExtendedFunction() >= 0x80000001 {
_, _, c, d := cpuid(0x80000001)
if (c & (1 << 5)) != 0 {
rval |= lzcnt
rval |= popcnt
}
if (d & (1 << 31)) != 0 {
rval |= amd3dnow
}
if (d & (1 << 30)) != 0 {
rval |= amd3dnowext
}
if (d & (1 << 23)) != 0 {
rval |= mmx
}
if (d & (1 << 22)) != 0 {
rval |= mmxext
}
if (c & (1 << 6)) != 0 {
rval |= sse4a
}
if d&(1<<20) != 0 {
rval |= nx
}
if d&(1<<27) != 0 {
rval |= rdtscp
}
/* Allow for selectively disabling SSE2 functions on AMD processors
with SSE2 support but not SSE4a. This includes Athlon64, some
Opteron, and some Sempron processors. MMX, SSE, or 3DNow! are faster
than SSE2 often enough to utilize this special-case flag.
AV_CPU_FLAG_SSE2 and AV_CPU_FLAG_SSE2SLOW are both set in this case
so that SSE2 is used unless explicitly disabled by checking
AV_CPU_FLAG_SSE2SLOW. */
if vendorID() != intel &&
rval&sse2 != 0 && (c&0x00000040) == 0 {
rval |= sse2slow
}
/* XOP and FMA4 use the AVX instruction coding scheme, so they can't be
* used unless the OS has AVX support. */
if (rval & avx) != 0 {
if (c & 0x00000800) != 0 {
rval |= xop
}
if (c & 0x00010000) != 0 {
rval |= fma4
}
}
if vendorID() == intel {
family, model := familyModel()
if family == 6 && (model == 9 || model == 13 || model == 14) {
/* 6/9 (pentium-m "banias"), 6/13 (pentium-m "dothan"), and
* 6/14 (core1 "yonah") theoretically support sse2, but it's
* usually slower than mmx. */
if (rval & sse2) != 0 {
rval |= sse2slow
}
if (rval & sse3) != 0 {
rval |= sse3slow
}
}
/* The Atom processor has SSSE3 support, which is useful in many cases,
* but sometimes the SSSE3 version is slower than the SSE2 equivalent
* on the Atom, but is generally faster on other processors supporting
* SSSE3. This flag allows for selectively disabling certain SSSE3
* functions on the Atom. */
if family == 6 && model == 28 {
rval |= atom
}
}
}
return flags(rval)
}
func valAsString(values ...uint32) []byte {
r := make([]byte, 4*len(values))
for i, v := range values {
dst := r[i*4:]
dst[0] = byte(v & 0xff)
dst[1] = byte((v >> 8) & 0xff)
dst[2] = byte((v >> 16) & 0xff)
dst[3] = byte((v >> 24) & 0xff)
switch {
case dst[0] == 0:
return r[:i*4]
case dst[1] == 0:
return r[:i*4+1]
case dst[2] == 0:
return r[:i*4+2]
case dst[3] == 0:
return r[:i*4+3]
}
}
return r
}

42
vendor/github.com/klauspost/cpuid/private/cpuid_386.s generated vendored Normal file
View File

@@ -0,0 +1,42 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build 386,!gccgo
// func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuid(SB), 7, $0
XORL CX, CX
MOVL op+0(FP), AX
CPUID
MOVL AX, eax+4(FP)
MOVL BX, ebx+8(FP)
MOVL CX, ecx+12(FP)
MOVL DX, edx+16(FP)
RET
// func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuidex(SB), 7, $0
MOVL op+0(FP), AX
MOVL op2+4(FP), CX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func xgetbv(index uint32) (eax, edx uint32)
TEXT ·asmXgetbv(SB), 7, $0
MOVL index+0(FP), CX
BYTE $0x0f; BYTE $0x01; BYTE $0xd0 // XGETBV
MOVL AX, eax+4(FP)
MOVL DX, edx+8(FP)
RET
// func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
TEXT ·asmRdtscpAsm(SB), 7, $0
BYTE $0x0F; BYTE $0x01; BYTE $0xF9 // RDTSCP
MOVL AX, eax+0(FP)
MOVL BX, ebx+4(FP)
MOVL CX, ecx+8(FP)
MOVL DX, edx+12(FP)
RET

42
vendor/github.com/klauspost/cpuid/private/cpuid_amd64.s generated vendored Normal file
View File

@@ -0,0 +1,42 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
//+build amd64,!gccgo
// func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuid(SB), 7, $0
XORQ CX, CX
MOVL op+0(FP), AX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
TEXT ·asmCpuidex(SB), 7, $0
MOVL op+0(FP), AX
MOVL op2+4(FP), CX
CPUID
MOVL AX, eax+8(FP)
MOVL BX, ebx+12(FP)
MOVL CX, ecx+16(FP)
MOVL DX, edx+20(FP)
RET
// func asmXgetbv(index uint32) (eax, edx uint32)
TEXT ·asmXgetbv(SB), 7, $0
MOVL index+0(FP), CX
BYTE $0x0f; BYTE $0x01; BYTE $0xd0 // XGETBV
MOVL AX, eax+8(FP)
MOVL DX, edx+12(FP)
RET
// func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
TEXT ·asmRdtscpAsm(SB), 7, $0
BYTE $0x0F; BYTE $0x01; BYTE $0xF9 // RDTSCP
MOVL AX, eax+0(FP)
MOVL BX, ebx+4(FP)
MOVL CX, ecx+8(FP)
MOVL DX, edx+12(FP)
RET

17
vendor/github.com/klauspost/cpuid/private/cpuid_detect_intel.go generated vendored Normal file
View File

@@ -0,0 +1,17 @@
// Copyright (c) 2015 Klaus Post, released under MIT License. See LICENSE file.
// +build 386,!gccgo amd64,!gccgo
package cpuid
func asmCpuid(op uint32) (eax, ebx, ecx, edx uint32)
func asmCpuidex(op, op2 uint32) (eax, ebx, ecx, edx uint32)
func asmXgetbv(index uint32) (eax, edx uint32)
func asmRdtscpAsm() (eax, ebx, ecx, edx uint32)
func initCPU() {
cpuid = asmCpuid
cpuidex = asmCpuidex
xgetbv = asmXgetbv
rdtscpAsm = asmRdtscpAsm
}

Some files were not shown because too many files have changed in this diff Show More