Fixed an off-by-one bug with managing datafiles (#31 )

Update README.md
2026-04-03 03:29:11 -07:00 · 2019-07-29 23:49:37 +10:00 · 2019-07-27 07:57:38 +10:00 · 2019-07-27 07:56:34 +10:00 · 2019-07-27 07:52:25 +10:00 · 2019-07-25 08:31:44 +10:00
33 changed files with 1363 additions and 591 deletions
--- a/.drone.yml
+++ b/.drone.yml
@@ -5,18 +5,10 @@ steps:
  - name: build
    image: golang:latest
    commands:
-      - go test -v -short -cover -coverprofile=coverage.txt ./...
+      - go test -v -short -cover -coverprofile=coverage.txt -coverpkg=$(go list) .

  - name: coverage
    image: plugins/codecov
    settings:
      token:
        from_secret: codecov-token
-
-  - name: notify
-    image: plugins/webhook
-    urls: https://msgbus.mills.io/ci.mills.io
-    when:
-      status:
-        - success
-        - failure
--- a/.goreleaser.yml
+++ b/.goreleaser.yml
@@ -3,14 +3,14 @@ builds:
    binary: bitcask
    main: ./cmd/bitcask
    flags: -tags "static_build"
-    ldflags: -w -X .Version={{.Version}} -X .Commit={{.Commit}}
+    ldflags: -w -X github.com/prologic/bitcask/internal.Version={{.Version}} -X github.com/prologic/bitcask/internal.Commit={{.Commit}}
    env:
      - CGO_ENABLED=0
  - 
    binary: bitcaskd
    main: ./cmd/bitcaskd
    flags: -tags "static_build"
-    ldflags: -w -X .Version={{.Version}} -X .Commit={{.Commit}}
+    ldflags: -w -X github.com/prologic/bitcask/internal.Version={{.Version}} -X github.com/prologic/bitcask/internal.Commit={{.Commit}}
    env:
      - CGO_ENABLED=0
 sign:
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -0,0 +1,8 @@
+# Contributing
+
+No preference. If you know hot to use Github and contributed to open source projects before then:
+
+* File an issue
+* Submit a pull request
+* File an issue + Submit a pull request
+* Use this project somewhere :)
--- a/14
+++ b/14
@@ -0,0 +1,14 @@
+# Build
+FROM prologic/go-builder:latest AS build
+
+# Runtime
+FROM alpine
+
+COPY --from=build /src/bitcaskd /bitcaskd
+
+EXPOSE 6379/tcp
+
+VOLUME /data
+
+ENTRYPOINT ["/bitcaskd"]
+CMD ["/data"]
--- a/15
+++ b/15
@@ -13,11 +13,11 @@ dev: build
 build: clean generate
 	@go build \
 		-tags "netgo static_build" -installsuffix netgo \
-		-ldflags "-w -X $(shell go list).Version=$(VERSION) -X $(shell go list).Commit=$(COMMIT)" \
+		-ldflags "-w -X $(shell go list)/internal.Version=$(VERSION) -X $(shell go list)/internal.Commit=$(COMMIT)" \
 		./cmd/bitcask/...
 	@go build \
 		-tags "netgo static_build" -installsuffix netgo \
-		-ldflags "-w -X $(shell go list).Version=$(VERSION) -X $(shell go list).Commit=$(COMMIT)" \
+		-ldflags "-w -X $(shell go list)/internal.Version=$(VERSION) -X $(shell go list)/internal.Commit=$(COMMIT)" \
 		./cmd/bitcaskd/...

 generate:
@@ -25,6 +25,7 @@ generate:

 install: build
 	@go install ./cmd/bitcask/...
+	@go install ./cmd/bitcaskd/...

 image:
 	@docker build -t prologic/bitcask .
@@ -33,13 +34,17 @@ release:
 	@./tools/release.sh

 profile: build
-	@go test -cpuprofile cpu.prof -memprofile mem.prof -v -bench ./...
+	@go test -cpuprofile cpu.prof -memprofile mem.prof -v -bench .

 bench: build
-	@go test -v -benchmem -bench=. ./...
+	@go test -v -benchmem -bench=. .

 test: build
-	@go test -v -cover -coverprofile=coverage.txt -covermode=atomic -coverpkg=./... -race ./...
+	@go test -v \
+		-cover -coverprofile=coverage.txt -covermode=atomic \
+		-coverpkg=$(shell go list) \
+		-race \
+		.

 clean:
 	@git clean -f -d -X
--- a/README.md
+++ b/README.md
@@ -5,17 +5,48 @@
 [![Go Report Card](https://goreportcard.com/badge/prologic/bitcask)](https://goreportcard.com/report/prologic/bitcask)
 [![GoDoc](https://godoc.org/github.com/prologic/bitcask?status.svg)](https://godoc.org/github.com/prologic/bitcask) 
 [![Sourcegraph](https://sourcegraph.com/github.com/prologic/bitcask/-/badge.svg)](https://sourcegraph.com/github.com/prologic/bitcask?badge)
+[![Github all releases](https://img.shields.io/github/downloads/prologic/bitcask/total.svg)](https://github.com/prologic/bitcask/releases)
+[![GitHub license](https://img.shields.io/github/license/prologic/bitcask.svg)](https://github.com/prologic/bitcask)

-A Bitcask (LSM+WAL) Key/Value Store written in Go.
+[![](https://images.microbadger.com/badges/version/prologic/bitcask.svg)](https://microbadger.com/images/prologic/bitcask)
+[![](https://images.microbadger.com/badges/image/prologic/bitcask.svg)](https://microbadger.com/images/prologic/bitcask)
+
+A high performance Key/Value store written in [Go](https://golang.org) with a predictable read/write performance and high throughput. Uses a [Bitcask](https://en.wikipedia.org/wiki/Bitcask) on-disk layout (LSM+WAL) similar to [Riak](https://riak.com/). 🗃️
+
+For a more feature-complete Redis-compatible server, distributed key/value store have a look at [Bitraft](https://github.com/prologic/bitraft) which uses this library as its backend. Use [Bitcask](https://github.com/prologic/bitcask) as a starting point or if you want to embed in your application, use [Bitraft](https://github.com/prologic/bitraft) if you need a complete server/client solution with high availability with a Redis-compatible API.

 ## Features

-* Embeddable
-* Builtin CLI
-* Builtin Redis-compatible server
+* Embeddable (`import "github.com/prologic/bitcask"`)
+* Builtin CLI (`bitcask`)
+* Builtin Redis-compatible server (`bitcaskd`)
 * Predictable read/write performance
 * Low latecny
-* High throughput (See: [Performance](README.md#Performance)
+* High throughput (See: [Performance](README.md#Performance) )
+
+## Development
+
+1. Get the source
+
+```#!bash
+$ git clone https://github.com/prologic/bitcask.git
+```
+
+2. Install required tools
+
+This library uses [Protobuf](https://github.com/protocolbuffers/protobuf) to serialize data on disk. Please follow the
+instructions for installing `protobuf` on your system. You will also need the
+following Go libraries/tools to generate Go code from Protobuf defs:
+
+3. Build the project
+
+```#!bash
+$ make
+```
+
+This will invoke `go generate` and `go build`.
+
+- [protoc-gen-go](https://github.com/golang/protobuf)

 ## Install

@@ -34,16 +65,13 @@ $ go get github.com/prologic/bitcask
 ```#!go
 package main

-import (
-    "log"
-
-    "github.com/prologic/bitcask"
-)
+import "github.com/prologic/bitcask"

 func main() {
    db, _ := bitcask.Open("/tmp/db")
-    db.Set("Hello", []byte("World"))
-    db.Close()
+    defer db.Close()
+    db.Put("Hello", []byte("World"))
+    val, _ := db.Get("Hello")
 }
 ```

@@ -90,6 +118,15 @@ QUIT
 Connection closed by foreign host.
 ```

+## Docker
+
+You can also use the [Bitcask Docker Image](https://cloud.docker.com/u/prologic/repository/docker/prologic/bitcask):
+
+```#!bash
+$ docker pull prologic/bitcask
+$ docker run -d -p 6379:6379 prologic/bitcask
+```
+
 ## Performance

 Benchmarks run on a 11" Macbook with a 1.4Ghz Intel Core i7:
@@ -97,30 +134,33 @@ Benchmarks run on a 11" Macbook with a 1.4Ghz Intel Core i7:
 ```
 $ make bench
 ...
-BenchmarkGet/128B-4         	  200000	      5780 ns/op	     400 B/op	       5 allocs/op
-BenchmarkGet/256B-4         	  200000	      6138 ns/op	     656 B/op	       5 allocs/op
-BenchmarkGet/512B-4         	  200000	      5967 ns/op	    1200 B/op	       5 allocs/op
-BenchmarkGet/1K-4           	  200000	      6290 ns/op	    2288 B/op	       5 allocs/op
-BenchmarkGet/2K-4           	  200000	      6293 ns/op	    4464 B/op	       5 allocs/op
-BenchmarkGet/4K-4           	  200000	      7673 ns/op	    9072 B/op	       5 allocs/op
-BenchmarkGet/8K-4           	  200000	     10373 ns/op	   17776 B/op	       5 allocs/op
-BenchmarkGet/16K-4          	  100000	     14227 ns/op	   34928 B/op	       5 allocs/op
-BenchmarkGet/32K-4          	  100000	     25953 ns/op	   73840 B/op	       5 allocs/op
-BenchmarkPut/128B-4         	  100000	     17353 ns/op	     680 B/op	       5 allocs/op
-BenchmarkPut/256B-4         	  100000	     18620 ns/op	     808 B/op	       5 allocs/op
-BenchmarkPut/512B-4         	  100000	     19068 ns/op	    1096 B/op	       5 allocs/op
-BenchmarkPut/1K-4           	  100000	     23738 ns/op	    1673 B/op	       5 allocs/op
-BenchmarkPut/2K-4           	   50000	     25118 ns/op	    2826 B/op	       5 allocs/op
-BenchmarkPut/4K-4           	   50000	     44605 ns/op	    5389 B/op	       5 allocs/op
-BenchmarkPut/8K-4           	   30000	     55237 ns/op	   10001 B/op	       5 allocs/op
-BenchmarkPut/16K-4          	   20000	     78966 ns/op	   18972 B/op	       5 allocs/op
-BenchmarkPut/32K-4          	   10000	    116253 ns/op	   41520 B/op	       5 allocs/op
+BenchmarkGet/128B-4         	  300000	      4071 ns/op	  31.43 MB/s	     608 B/op	       7 allocs/op
+BenchmarkGet/256B-4         	  300000	      4700 ns/op	  54.46 MB/s	     992 B/op	       7 allocs/op
+BenchmarkGet/512B-4         	  300000	      4915 ns/op	 104.17 MB/s	    1824 B/op	       7 allocs/op
+BenchmarkGet/1K-4           	  200000	      5064 ns/op	 202.20 MB/s	    3488 B/op	       7 allocs/op
+BenchmarkGet/2K-4           	  200000	      6276 ns/op	 326.31 MB/s	    6816 B/op	       7 allocs/op
+BenchmarkGet/4K-4           	  200000	      8960 ns/op	 457.11 MB/s	   13984 B/op	       7 allocs/op
+BenchmarkGet/8K-4           	  100000	     12465 ns/op	 657.16 MB/s	   27296 B/op	       7 allocs/op
+BenchmarkGet/16K-4          	  100000	     19233 ns/op	 851.84 MB/s	   53408 B/op	       7 allocs/op
+BenchmarkGet/32K-4          	   50000	     33106 ns/op	 989.77 MB/s	  114848 B/op	       7 allocs/op
+
+BenchmarkPut/128B-4         	  100000	     13659 ns/op	   9.37 MB/s	     409 B/op	       6 allocs/op
+BenchmarkPut/256B-4         	  100000	     14854 ns/op	  17.23 MB/s	     539 B/op	       6 allocs/op
+BenchmarkPut/512B-4         	  100000	     20823 ns/op	  24.59 MB/s	     829 B/op	       6 allocs/op
+BenchmarkPut/1K-4           	   50000	     28086 ns/op	  36.46 MB/s	    1411 B/op	       6 allocs/op
+BenchmarkPut/2K-4           	   30000	     40797 ns/op	  50.20 MB/s	    2574 B/op	       6 allocs/op
+BenchmarkPut/4K-4           	   20000	     75518 ns/op	  54.24 MB/s	    5155 B/op	       6 allocs/op
+BenchmarkPut/8K-4           	   10000	    122544 ns/op	  66.85 MB/s	    9811 B/op	       6 allocs/op
+BenchmarkPut/16K-4          	   10000	    201167 ns/op	  81.44 MB/s	   18851 B/op	       6 allocs/op
+BenchmarkPut/32K-4          	    5000	    350850 ns/op	  93.40 MB/s	   41565 B/op	       7 allocs/op
+
+BenchmarkScan-4             	 1000000	      1867 ns/op	     493 B/op	      25 allocs/op
 ```

 For 128B values:

-* ~180,000 reads/sec
-*  ~60,000 writes/sec
+* ~400,000 reads/sec
+* ~130,000 writes/sec

 The full benchmark above shows linear performance as you increase key/value sizes.

--- a/bitcask.go
+++ b/bitcask.go
@@ -2,98 +2,204 @@ package bitcask

 import (
 	"errors"
-	"fmt"
+	"hash/crc32"
 	"io"
 	"io/ioutil"
 	"os"
+	"path"
 	"path/filepath"
-	"sort"
-	"strconv"
-	"strings"
-	"time"
+	"sync"

 	"github.com/gofrs/flock"
-)
+	"github.com/prologic/trie"

-const (
-	DefaultMaxDatafileSize = 1 << 20 // 1MB
+	"github.com/prologic/bitcask/internal"
 )

 var (
-	ErrKeyNotFound       = errors.New("error: key not found")
-	ErrCannotAcquireLock = errors.New("error: cannot acquire lock")
+	// ErrKeyNotFound is the error returned when a key is not found
+	ErrKeyNotFound = errors.New("error: key not found")
+
+	// ErrKeyTooLarge is the error returned for a key that exceeds the
+	// maximum allowed key size (configured with WithMaxKeySize).
+	ErrKeyTooLarge = errors.New("error: key too large")
+
+	// ErrValueTooLarge is the error returned for a value that exceeds the
+	// maximum allowed value size (configured with WithMaxValueSize).
+	ErrValueTooLarge = errors.New("error: value too large")
+
+	// ErrChecksumFailed is the error returned if a key/value retrieved does
+	// not match its CRC checksum
+	ErrChecksumFailed = errors.New("error: checksum failed")
+
+	// ErrDatabaseLocked is the error returned if the database is locked
+	// (typically opened by another process)
+	ErrDatabaseLocked = errors.New("error: database locked")
 )

+// Bitcask is a struct that represents a on-disk LSM and WAL data structure
+// and in-memory hash of key/value pairs as per the Bitcask paper and seen
+// in the Riak database.
 type Bitcask struct {
+	mu sync.RWMutex
+
 	*flock.Flock

+	config    *config
+	options   []Option
 	path      string
-	curr      *Datafile
-	keydir    *Keydir
-	datafiles []*Datafile
-
-	maxDatafileSize int64
+	curr      *internal.Datafile
+	keydir    *internal.Keydir
+	datafiles map[int]*internal.Datafile
+	trie      *trie.Trie
 }

+// Stats is a struct returned by Stats() on an open Bitcask instance
+type Stats struct {
+	Datafiles int
+	Keys      int
+	Size      int64
+}
+
+// Stats returns statistics about the database including the number of
+// data files, keys and overall size on disk of the data
+func (b *Bitcask) Stats() (stats Stats, err error) {
+	var size int64
+
+	size, err = internal.DirSize(b.path)
+	if err != nil {
+		return
+	}
+
+	stats.Datafiles = len(b.datafiles)
+	stats.Keys = b.keydir.Len()
+	stats.Size = size
+
+	return
+}
+
+// Close closes the database and removes the lock. It is important to call
+// Close() as this is the only way to cleanup the lock held by the open
+// database.
 func (b *Bitcask) Close() error {
 	defer func() {
 		b.Flock.Unlock()
+		os.Remove(b.Flock.Path())
 	}()

-	for _, df := range b.datafiles {
-		df.Close()
+	if err := b.keydir.Save(path.Join(b.path, "index")); err != nil {
+		return err
 	}
+
+	for _, df := range b.datafiles {
+		if err := df.Close(); err != nil {
+			return err
+		}
+	}
+
 	return b.curr.Close()
 }

+// Sync flushes all buffers to disk ensuring all data is written
 func (b *Bitcask) Sync() error {
 	return b.curr.Sync()
 }

+// Get retrieves the value of the given key. If the key is not found or an/I/O
+// error occurs a null byte slice is returned along with the error.
 func (b *Bitcask) Get(key string) ([]byte, error) {
-	var df *Datafile
+	var df *internal.Datafile

 	item, ok := b.keydir.Get(key)
 	if !ok {
 		return nil, ErrKeyNotFound
 	}

-	if item.FileID == b.curr.id {
+	if item.FileID == b.curr.FileID() {
 		df = b.curr
 	} else {
 		df = b.datafiles[item.FileID]
 	}

-	e, err := df.ReadAt(item.Index)
+	e, err := df.ReadAt(item.Offset, item.Size)
 	if err != nil {
 		return nil, err
 	}

+	checksum := crc32.ChecksumIEEE(e.Value)
+	if checksum != e.Checksum {
+		return nil, ErrChecksumFailed
+	}
+
 	return e.Value, nil
 }

+// Has returns true if the key exists in the database, false otherwise.
+func (b *Bitcask) Has(key string) bool {
+	_, ok := b.keydir.Get(key)
+	return ok
+}
+
+// Put stores the key and value in the database.
 func (b *Bitcask) Put(key string, value []byte) error {
-	index, err := b.put(key, value)
+	if len(key) > b.config.maxKeySize {
+		return ErrKeyTooLarge
+	}
+	if len(value) > b.config.maxValueSize {
+		return ErrValueTooLarge
+	}
+
+	offset, n, err := b.put(key, value)
 	if err != nil {
 		return err
 	}

-	b.keydir.Add(key, b.curr.id, index, time.Now().Unix())
+	item := b.keydir.Add(key, b.curr.FileID(), offset, n)
+	b.trie.Add(key, item)

 	return nil
 }

+// Delete deletes the named key. If the key doesn't exist or an I/O error
+// occurs the error is returned.
 func (b *Bitcask) Delete(key string) error {
-	_, err := b.put(key, []byte{})
+	_, _, err := b.put(key, []byte{})
 	if err != nil {
 		return err
 	}

 	b.keydir.Delete(key)
+	b.trie.Remove(key)

 	return nil
 }

+// Scan performs a prefix scan of keys matching the given prefix and calling
+// the function `f` with the keys found. If the function returns an error
+// no further keys are processed and the first error returned.
+func (b *Bitcask) Scan(prefix string, f func(key string) error) error {
+	keys := b.trie.PrefixSearch(prefix)
+	for _, key := range keys {
+		if err := f(key); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+// Len returns the total number of keys in the database
+func (b *Bitcask) Len() int {
+	return b.keydir.Len()
+}
+
+// Keys returns all keys in the database as a channel of string(s)
+func (b *Bitcask) Keys() chan string {
+	return b.keydir.Keys()
+}
+
+// Fold iterates over all keys in the database calling the function `f` for
+// each key. If the function returns an error, no further keys are processed
+// and the error returned.
 func (b *Bitcask) Fold(f func(key string) error) error {
 	for key := range b.keydir.Keys() {
 		if err := f(key); err != nil {
@@ -103,231 +209,82 @@ func (b *Bitcask) Fold(f func(key string) error) error {
 	return nil
 }

-func (b *Bitcask) put(key string, value []byte) (int64, error) {
-	size, err := b.curr.Size()
-	if err != nil {
-		return -1, err
-	}
+func (b *Bitcask) put(key string, value []byte) (int64, int64, error) {
+	b.mu.Lock()
+	defer b.mu.Unlock()

-	if size >= b.maxDatafileSize {
+	size := b.curr.Size()
+	if size >= int64(b.config.maxDatafileSize) {
 		err := b.curr.Close()
 		if err != nil {
-			return -1, err
+			return -1, 0, err
 		}

-		df, err := NewDatafile(b.path, b.curr.id, true)
-		b.datafiles = append(b.datafiles, df)
+		id := b.curr.FileID()

-		id := b.curr.id + 1
-		curr, err := NewDatafile(b.path, id, false)
+		df, err := internal.NewDatafile(b.path, id, true)
 		if err != nil {
-			return -1, err
+			return -1, 0, err
+		}
+
+		b.datafiles[id] = df
+
+		id = b.curr.FileID() + 1
+		curr, err := internal.NewDatafile(b.path, id, false)
+		if err != nil {
+			return -1, 0, err
 		}
 		b.curr = curr
 	}

-	e := NewEntry(key, value)
+	e := internal.NewEntry(key, value)
 	return b.curr.Write(e)
 }

-func (b *Bitcask) setMaxDatafileSize(size int64) error {
-	b.maxDatafileSize = size
-	return nil
-}
+func (b *Bitcask) reopen() error {
+	b.mu.Lock()
+	defer b.mu.Unlock()

-func WithMaxDatafileSize(size int64) func(*Bitcask) error {
-	return func(b *Bitcask) error {
-		return b.setMaxDatafileSize(size)
-	}
-}
-
-func getDatafiles(path string) ([]string, error) {
-	fns, err := filepath.Glob(fmt.Sprintf("%s/*.data", path))
-	if err != nil {
-		return nil, err
-	}
-	sort.Strings(fns)
-	return fns, nil
-}
-
-func parseIds(fns []string) ([]int, error) {
-	var ids []int
-	for _, fn := range fns {
-		fn = filepath.Base(fn)
-		ext := filepath.Ext(fn)
-		if ext != ".data" {
-			continue
-		}
-		id, err := strconv.ParseInt(strings.TrimSuffix(fn, ext), 10, 32)
-		if err != nil {
-			return nil, err
-		}
-		ids = append(ids, int(id))
-	}
-	sort.Ints(ids)
-	return ids, nil
-}
-
-func Merge(path string, force bool) error {
-	fns, err := getDatafiles(path)
+	fns, err := internal.GetDatafiles(b.path)
 	if err != nil {
 		return err
 	}

-	ids, err := parseIds(fns)
+	ids, err := internal.ParseIds(fns)
 	if err != nil {
 		return err
 	}

-	// Do not merge if we only have 1 Datafile
-	if len(ids) <= 1 {
-		return nil
-	}
+	datafiles := make(map[int]*internal.Datafile, len(ids))

-	// Don't merge the Active Datafile (the last one)
-	fns = fns[:len(fns)-1]
-	ids = ids[:len(ids)-1]
-
-	temp, err := ioutil.TempDir("", "bitcask")
-	if err != nil {
-		return err
-	}
-
-	for i, fn := range fns {
-		// Don't merge Datafiles whose .hint files we've already generated
-		// (they are already merged); unless we set the force flag to true
-		// (forcing a re-merge).
-		if filepath.Ext(fn) == ".hint" && !force {
-			// Already merged
-			continue
-		}
-
-		id := ids[i]
-
-		keydir := NewKeydir()
-
-		df, err := NewDatafile(path, id, true)
+	for _, id := range ids {
+		df, err := internal.NewDatafile(b.path, id, true)
 		if err != nil {
 			return err
 		}
-		defer df.Close()
+		datafiles[id] = df
+	}

-		for {
-			e, err := df.Read()
-			if err != nil {
-				if err == io.EOF {
-					break
-				}
-				return err
-			}
+	keydir := internal.NewKeydir()
+	trie := trie.New()

-			// Tombstone value  (deleted key)
-			if len(e.Value) == 0 {
-				keydir.Delete(e.Key)
-				continue
-			}
-
-			keydir.Add(e.Key, ids[i], e.Index, e.Timestamp)
-		}
-
-		tempdf, err := NewDatafile(temp, id, false)
-		if err != nil {
+	if internal.Exists(path.Join(b.path, "index")) {
+		if err := keydir.Load(path.Join(b.path, "index")); err != nil {
 			return err
 		}
-		defer tempdf.Close()
-
 		for key := range keydir.Keys() {
 			item, _ := keydir.Get(key)
-			e, err := df.ReadAt(item.Index)
-			if err != nil {
-				return err
-			}
-
-			_, err = tempdf.Write(e)
-			if err != nil {
-				return err
-			}
+			trie.Add(key, item)
 		}
-
-		err = tempdf.Close()
-		if err != nil {
-			return err
-		}
-
-		err = df.Close()
-		if err != nil {
-			return err
-		}
-
-		err = os.Rename(tempdf.Name(), df.Name())
-		if err != nil {
-			return err
-		}
-
-		hint := strings.TrimSuffix(df.Name(), ".data") + ".hint"
-		err = keydir.Save(hint)
-		if err != nil {
-			return err
-		}
-	}
-
-	return nil
-}
-
-func Open(path string, options ...func(*Bitcask) error) (*Bitcask, error) {
-	if err := os.MkdirAll(path, 0755); err != nil {
-		return nil, err
-	}
-
-	err := Merge(path, false)
-	if err != nil {
-		return nil, err
-	}
-
-	fns, err := getDatafiles(path)
-	if err != nil {
-		return nil, err
-	}
-
-	ids, err := parseIds(fns)
-	if err != nil {
-		return nil, err
-	}
-
-	keydir := NewKeydir()
-	var datafiles []*Datafile
-
-	for i, fn := range fns {
-		df, err := NewDatafile(path, ids[i], true)
-		if err != nil {
-			return nil, err
-		}
-		datafiles = append(datafiles, df)
-
-		if filepath.Ext(fn) == ".hint" {
-			f, err := os.Open(filepath.Join(path, fn))
-			if err != nil {
-				return nil, err
-			}
-			defer f.Close()
-
-			hint, err := NewKeydirFromBytes(f)
-			if err != nil {
-				return nil, err
-			}
-
-			for key := range hint.Keys() {
-				item, _ := hint.Get(key)
-				keydir.Add(key, item.FileID, item.Index, item.Timestamp)
-			}
-		} else {
+	} else {
+		for i, df := range datafiles {
 			for {
-				e, err := df.Read()
+				e, n, err := df.Read()
 				if err != nil {
 					if err == io.EOF {
 						break
 					}
-					return nil, err
+					return err
 				}

 				// Tombstone value  (deleted key)
@@ -336,7 +293,8 @@ func Open(path string, options ...func(*Bitcask) error) (*Bitcask, error) {
 					continue
 				}

-				keydir.Add(e.Key, ids[i], e.Index, e.Timestamp)
+				item := keydir.Add(e.Key, ids[i], e.Offset, n)
+				trie.Add(e.Key, item)
 			}
 		}
 	}
@@ -346,24 +304,118 @@ func Open(path string, options ...func(*Bitcask) error) (*Bitcask, error) {
 		id = ids[(len(ids) - 1)]
 	}

-	curr, err := NewDatafile(path, id, false)
+	curr, err := internal.NewDatafile(b.path, id, false)
 	if err != nil {
+		return err
+	}
+
+	b.curr = curr
+	b.datafiles = datafiles
+
+	b.keydir = keydir
+
+	b.trie = trie
+
+	return nil
+}
+
+// Merge merges all datafiles in the database. Old keys are squashed
+// and deleted keys removes. Duplicate key/value pairs are also removed.
+// Call this function periodically to reclaim disk space.
+func (b *Bitcask) Merge() error {
+	// Temporary merged database path
+	temp, err := ioutil.TempDir(b.path, "merge")
+	if err != nil {
+		return err
+	}
+	defer os.RemoveAll(temp)
+
+	// Create a merged database
+	mdb, err := Open(temp, b.options...)
+	if err != nil {
+		return err
+	}
+
+	// Rewrite all key/value pairs into merged database
+	// Doing this automatically strips deleted keys and
+	// old key/value pairs
+	err = b.Fold(func(key string) error {
+		value, err := b.Get(key)
+		if err != nil {
+			return err
+		}
+
+		if err := mdb.Put(key, value); err != nil {
+			return err
+		}
+
+		return nil
+	})
+	if err != nil {
+		return err
+	}
+
+	err = mdb.Close()
+	if err != nil {
+		return err
+	}
+
+	// Close the database
+	err = b.Close()
+	if err != nil {
+		return err
+	}
+
+	// Remove all data files
+	files, err := ioutil.ReadDir(b.path)
+	if err != nil {
+		return err
+	}
+	for _, file := range files {
+		if !file.IsDir() {
+			err := os.RemoveAll(path.Join([]string{b.path, file.Name()}...))
+			if err != nil {
+				return err
+			}
+		}
+	}
+
+	// Rename all merged data files
+	files, err = ioutil.ReadDir(mdb.path)
+	if err != nil {
+		return err
+	}
+	for _, file := range files {
+		err := os.Rename(
+			path.Join([]string{mdb.path, file.Name()}...),
+			path.Join([]string{b.path, file.Name()}...),
+		)
+		if err != nil {
+			return err
+		}
+	}
+
+	// And finally reopen the database
+	return b.reopen()
+}
+
+// Open opens the database at the given path with optional options.
+// Options can be provided with the `WithXXX` functions that provide
+// configuration options as functions.
+func Open(path string, options ...Option) (*Bitcask, error) {
+	if err := os.MkdirAll(path, 0755); err != nil {
 		return nil, err
 	}

 	bitcask := &Bitcask{
-		Flock:     flock.New(filepath.Join(path, "lock")),
-		path:      path,
-		curr:      curr,
-		keydir:    keydir,
-		datafiles: datafiles,
-
-		maxDatafileSize: DefaultMaxDatafileSize,
+		Flock:   flock.New(filepath.Join(path, "lock")),
+		config:  newDefaultConfig(),
+		options: options,
+		path:    path,
 	}

-	for _, option := range options {
-		err = option(bitcask)
-		if err != nil {
+	for _, opt := range options {
+		if err := opt(bitcask.config); err != nil {
 			return nil, err
 		}
 	}
@@ -374,8 +426,25 @@ func Open(path string, options ...func(*Bitcask) error) (*Bitcask, error) {
 	}

 	if !locked {
-		return nil, ErrCannotAcquireLock
+		return nil, ErrDatabaseLocked
+	}
+
+	if err := bitcask.reopen(); err != nil {
+		return nil, err
 	}

 	return bitcask, nil
 }
+
+// Merge calls Bitcask.Merge()
+// XXX: Deprecated; Please use the `.Merge()` method
+// XXX: This is only kept here for backwards compatibility
+//      it will be removed in future releases at some point
+func Merge(path string, force bool) error {
+	db, err := Open(path)
+	if err != nil {
+		return err
+	}
+
+	return db.Merge()
+}
--- a/bitcask_test.go
+++ b/bitcask_test.go
@@ -3,6 +3,9 @@ package bitcask
 import (
 	"fmt"
 	"io/ioutil"
+	"os"
+	"reflect"
+	"sort"
 	"strings"
 	"sync"
 	"testing"
@@ -38,12 +41,48 @@ func TestAll(t *testing.T) {
 		assert.Equal([]byte("bar"), val)
 	})

+	t.Run("Len", func(t *testing.T) {
+		assert.Equal(1, db.Len())
+	})
+
+	t.Run("Has", func(t *testing.T) {
+		assert.True(db.Has("foo"))
+	})
+
+	t.Run("Keys", func(t *testing.T) {
+		keys := make([]string, 0)
+		for key := range db.Keys() {
+			keys = append(keys, key)
+		}
+		assert.Equal([]string{"foo"}, keys)
+	})
+
+	t.Run("Fold", func(t *testing.T) {
+		var (
+			keys   []string
+			values [][]byte
+		)
+
+		err := db.Fold(func(key string) error {
+			value, err := db.Get(key)
+			if err != nil {
+				return err
+			}
+			keys = append(keys, key)
+			values = append(values, value)
+			return nil
+		})
+		assert.NoError(err)
+		assert.Equal([]string{"foo"}, keys)
+		assert.Equal([][]byte{[]byte("bar")}, values)
+	})
+
 	t.Run("Delete", func(t *testing.T) {
 		err := db.Delete("foo")
 		assert.NoError(err)
 		_, err = db.Get("foo")
 		assert.Error(err)
-		assert.Equal(err.Error(), "error: key not found")
+		assert.Equal(ErrKeyNotFound, err)
 	})

 	t.Run("Sync", func(t *testing.T) {
@@ -90,7 +129,7 @@ func TestDeletedKeys(t *testing.T) {
 			assert.NoError(err)
 			_, err = db.Get("foo")
 			assert.Error(err)
-			assert.Equal("error: key not found", err.Error())
+			assert.Equal(ErrKeyNotFound, err)
 		})

 		t.Run("Sync", func(t *testing.T) {
@@ -118,7 +157,7 @@ func TestDeletedKeys(t *testing.T) {
 		t.Run("Get", func(t *testing.T) {
 			_, err = db.Get("foo")
 			assert.Error(err)
-			assert.Equal("error: key not found", err.Error())
+			assert.Equal(ErrKeyNotFound, err)
 		})

 		t.Run("Close", func(t *testing.T) {
@@ -128,38 +167,83 @@ func TestDeletedKeys(t *testing.T) {
 	})
 }

-func TestMerge(t *testing.T) {
+func TestMaxKeySize(t *testing.T) {
+	assert := assert.New(t)
+
+	testdir, err := ioutil.TempDir("", "bitcask")
+	assert.NoError(err)
+
+	var db *Bitcask
+
+	t.Run("Open", func(t *testing.T) {
+		db, err = Open(testdir, WithMaxKeySize(16))
+		assert.NoError(err)
+	})
+
+	t.Run("Put", func(t *testing.T) {
+		key := strings.Repeat(" ", 17)
+		value := []byte("foobar")
+		err = db.Put(key, value)
+		assert.Error(err)
+		assert.Equal(ErrKeyTooLarge, err)
+	})
+}
+
+func TestMaxValueSize(t *testing.T) {
+	assert := assert.New(t)
+
+	testdir, err := ioutil.TempDir("", "bitcask")
+	assert.NoError(err)
+
+	var db *Bitcask
+
+	t.Run("Open", func(t *testing.T) {
+		db, err = Open(testdir, WithMaxValueSize(16))
+		assert.NoError(err)
+	})
+
+	t.Run("Put", func(t *testing.T) {
+		key := "foo"
+		value := []byte(strings.Repeat(" ", 17))
+		err = db.Put(key, value)
+		assert.Error(err)
+		assert.Equal(ErrValueTooLarge, err)
+	})
+}
+
+func TestStats(t *testing.T) {
+	var (
+		db  *Bitcask
+		err error
+	)
+
 	assert := assert.New(t)

 	testdir, err := ioutil.TempDir("", "bitcask")
 	assert.NoError(err)

 	t.Run("Setup", func(t *testing.T) {
-		var (
-			db  *Bitcask
-			err error
-		)
-
 		t.Run("Open", func(t *testing.T) {
-			db, err = Open(testdir, WithMaxDatafileSize(1024))
+			db, err = Open(testdir)
 			assert.NoError(err)
 		})

 		t.Run("Put", func(t *testing.T) {
-			for i := 0; i < 1024; i++ {
-				err = db.Put(string(i), []byte(strings.Repeat(" ", 1024)))
-				assert.NoError(err)
-			}
+			err := db.Put("foo", []byte("bar"))
+			assert.NoError(err)
 		})

 		t.Run("Get", func(t *testing.T) {
-			for i := 0; i < 32; i++ {
-				err = db.Put(string(i), []byte(strings.Repeat(" ", 1024)))
-				assert.NoError(err)
-				val, err := db.Get(string(i))
-				assert.NoError(err)
-				assert.Equal([]byte(strings.Repeat(" ", 1024)), val)
-			}
+			val, err := db.Get("foo")
+			assert.NoError(err)
+			assert.Equal([]byte("bar"), val)
+		})
+
+		t.Run("Stats", func(t *testing.T) {
+			stats, err := db.Stats()
+			assert.NoError(err)
+			assert.Equal(stats.Datafiles, 0)
+			assert.Equal(stats.Keys, 1)
 		})

 		t.Run("Sync", func(t *testing.T) {
@@ -172,26 +256,65 @@ func TestMerge(t *testing.T) {
 			assert.NoError(err)
 		})
 	})
+}

-	t.Run("Merge", func(t *testing.T) {
-		var (
-			db  *Bitcask
-			err error
-		)
+func TestMerge(t *testing.T) {
+	var (
+		db  *Bitcask
+		err error
+	)

+	assert := assert.New(t)
+
+	testdir, err := ioutil.TempDir("", "bitcask")
+	assert.NoError(err)
+
+	t.Run("Setup", func(t *testing.T) {
 		t.Run("Open", func(t *testing.T) {
-			db, err = Open(testdir)
+			db, err = Open(testdir, WithMaxDatafileSize(32))
 			assert.NoError(err)
 		})

-		t.Run("Get", func(t *testing.T) {
-			for i := 0; i < 32; i++ {
-				val, err := db.Get(string(i))
+		t.Run("Put", func(t *testing.T) {
+			err := db.Put("foo", []byte("bar"))
+			assert.NoError(err)
+		})
+
+		s1, err := db.Stats()
+		assert.NoError(err)
+		assert.Equal(0, s1.Datafiles)
+		assert.Equal(1, s1.Keys)
+
+		t.Run("Put", func(t *testing.T) {
+			for i := 0; i < 10; i++ {
+				err := db.Put("foo", []byte("bar"))
 				assert.NoError(err)
-				assert.Equal([]byte(strings.Repeat(" ", 1024)), val)
 			}
 		})

+		s2, err := db.Stats()
+		assert.NoError(err)
+		assert.Equal(5, s2.Datafiles)
+		assert.Equal(1, s2.Keys)
+		assert.True(s2.Size > s1.Size)
+
+		t.Run("Merge", func(t *testing.T) {
+			err := db.Merge()
+			assert.NoError(err)
+		})
+
+		s3, err := db.Stats()
+		assert.NoError(err)
+		assert.Equal(1, s3.Datafiles)
+		assert.Equal(1, s3.Keys)
+		assert.True(s3.Size > s1.Size)
+		assert.True(s3.Size < s2.Size)
+
+		t.Run("Sync", func(t *testing.T) {
+			err = db.Sync()
+			assert.NoError(err)
+		})
+
 		t.Run("Close", func(t *testing.T) {
 			err = db.Close()
 			assert.NoError(err)
@@ -241,10 +364,9 @@ func TestConcurrent(t *testing.T) {
 			wg := &sync.WaitGroup{}

 			go f(wg, 2)
-			wg.Add(1)
-
 			go f(wg, 3)
-			wg.Add(1)
+			go f(wg, 5)
+			wg.Add(3)

 			wg.Wait()
 		})
@@ -264,10 +386,9 @@ func TestConcurrent(t *testing.T) {
 			wg := &sync.WaitGroup{}

 			go f(wg, 100)
-			wg.Add(1)
-
 			go f(wg, 100)
-			wg.Add(1)
+			go f(wg, 100)
+			wg.Add(3)

 			wg.Wait()
 		})
@@ -279,6 +400,58 @@ func TestConcurrent(t *testing.T) {
 	})
 }

+func TestScan(t *testing.T) {
+	assert := assert.New(t)
+
+	testdir, err := ioutil.TempDir("", "bitcask")
+	assert.NoError(err)
+
+	var db *Bitcask
+
+	t.Run("Setup", func(t *testing.T) {
+		t.Run("Open", func(t *testing.T) {
+			db, err = Open(testdir)
+			assert.NoError(err)
+		})
+
+		t.Run("Put", func(t *testing.T) {
+			var items = map[string][]byte{
+				"1":     []byte("1"),
+				"2":     []byte("2"),
+				"3":     []byte("3"),
+				"food":  []byte("pizza"),
+				"foo":   []byte("foo"),
+				"fooz":  []byte("fooz ball"),
+				"hello": []byte("world"),
+			}
+			for k, v := range items {
+				err = db.Put(k, v)
+				assert.NoError(err)
+			}
+		})
+	})
+
+	t.Run("Scan", func(t *testing.T) {
+		var (
+			vals     []string
+			expected = []string{
+				"foo",
+				"fooz ball",
+				"pizza",
+			}
+		)
+
+		err = db.Scan("fo", func(key string) error {
+			val, err := db.Get(key)
+			assert.NoError(err)
+			vals = append(vals, string(val))
+			return nil
+		})
+		sort.Strings(vals)
+		assert.Equal(expected, vals)
+	})
+}
+
 func TestLocking(t *testing.T) {
 	assert := assert.New(t)

@@ -291,7 +464,7 @@ func TestLocking(t *testing.T) {

 	_, err = Open(testdir)
 	assert.Error(err)
-	assert.Equal("error: cannot acquire lock", err.Error())
+	assert.Equal(ErrDatabaseLocked, err)
 }

 type benchmarkTestCase struct {
@@ -300,11 +473,17 @@ type benchmarkTestCase struct {
 }

 func BenchmarkGet(b *testing.B) {
-	testdir, err := ioutil.TempDir("", "bitcask")
+	currentDir, err := os.Getwd()
 	if err != nil {
 		b.Fatal(err)
 	}

+	testdir, err := ioutil.TempDir(currentDir, "bitcask_bench")
+	if err != nil {
+		b.Fatal(err)
+	}
+	defer os.RemoveAll(testdir)
+
 	db, err := Open(testdir)
 	if err != nil {
 		b.Fatal(err)
@@ -325,6 +504,8 @@ func BenchmarkGet(b *testing.B) {

 	for _, tt := range tests {
 		b.Run(tt.name, func(b *testing.B) {
+			b.SetBytes(int64(tt.size))
+
 			key := "foo"
 			value := []byte(strings.Repeat(" ", tt.size))

@@ -348,11 +529,17 @@ func BenchmarkGet(b *testing.B) {
 }

 func BenchmarkPut(b *testing.B) {
-	testdir, err := ioutil.TempDir("", "bitcask")
+	currentDir, err := os.Getwd()
 	if err != nil {
 		b.Fatal(err)
 	}

+	testdir, err := ioutil.TempDir(currentDir, "bitcask_bench")
+	if err != nil {
+		b.Fatal(err)
+	}
+	defer os.RemoveAll(testdir)
+
 	db, err := Open(testdir)
 	if err != nil {
 		b.Fatal(err)
@@ -373,6 +560,8 @@ func BenchmarkPut(b *testing.B) {

 	for _, tt := range tests {
 		b.Run(tt.name, func(b *testing.B) {
+			b.SetBytes(int64(tt.size))
+
 			key := "foo"
 			value := []byte(strings.Repeat(" ", tt.size))
 			b.ResetTimer()
@@ -385,3 +574,56 @@ func BenchmarkPut(b *testing.B) {
 		})
 	}
 }
+
+func BenchmarkScan(b *testing.B) {
+	currentDir, err := os.Getwd()
+	if err != nil {
+		b.Fatal(err)
+	}
+
+	testdir, err := ioutil.TempDir(currentDir, "bitcask_bench")
+	if err != nil {
+		b.Fatal(err)
+	}
+	defer os.RemoveAll(testdir)
+
+	db, err := Open(testdir)
+	if err != nil {
+		b.Fatal(err)
+	}
+	defer db.Close()
+
+	var items = map[string][]byte{
+		"1":     []byte("1"),
+		"2":     []byte("2"),
+		"3":     []byte("3"),
+		"food":  []byte("pizza"),
+		"foo":   []byte("foo"),
+		"fooz":  []byte("fooz ball"),
+		"hello": []byte("world"),
+	}
+	for k, v := range items {
+		err := db.Put(k, v)
+		if err != nil {
+			b.Fatal(err)
+		}
+	}
+
+	var expected = []string{"foo", "food", "fooz"}
+
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		var keys []string
+		err = db.Scan("fo", func(key string) error {
+			keys = append(keys, key)
+			return nil
+		})
+		if err != nil {
+			b.Fatal(err)
+		}
+		sort.Strings(keys)
+		if !reflect.DeepEqual(expected, keys) {
+			b.Fatal(fmt.Errorf("expected keys=#%v got=%#v", expected, keys))
+		}
+	}
+}
--- a/cmd/bitcask/del.go
+++ b/cmd/bitcask/del.go
@@ -35,6 +35,7 @@ func del(path, key string) int {
 		log.WithError(err).Error("error opening database")
 		return 1
 	}
+	defer db.Close()

 	err = db.Delete(key)
 	if err != nil {
--- a/cmd/bitcask/get.go
+++ b/cmd/bitcask/get.go
@@ -36,6 +36,7 @@ func get(path, key string) int {
 		log.WithError(err).Error("error opening database")
 		return 1
 	}
+	defer db.Close()

 	value, err := db.Get(key)
 	if err != nil {
--- a/cmd/bitcask/keys.go
+++ b/cmd/bitcask/keys.go
@@ -34,6 +34,7 @@ func keys(path string) int {
 		log.WithError(err).Error("error opening database")
 		return 1
 	}
+	defer db.Close()

 	err = db.Fold(func(key string) error {
 		fmt.Printf("%s\n", key)
--- a/cmd/bitcask/merge.go
+++ b/cmd/bitcask/merge.go
@@ -20,28 +20,23 @@ keys.`,
 	Args: cobra.ExactArgs(0),
 	Run: func(cmd *cobra.Command, args []string) {
 		path := viper.GetString("path")
-		force, err := cmd.Flags().GetBool("force")
-		if err != nil {
-			log.WithError(err).Error("error parsing force flag")
-			os.Exit(1)
-		}

-		os.Exit(merge(path, force))
+		os.Exit(merge(path))
 	},
 }

 func init() {
 	RootCmd.AddCommand(mergeCmd)
-
-	mergeCmd.Flags().BoolP(
-		"force", "f", false,
-		"Force a re-merge even if .hint files exist",
-	)
 }

-func merge(path string, force bool) int {
-	err := bitcask.Merge(path, force)
+func merge(path string) int {
+	db, err := bitcask.Open(path)
 	if err != nil {
+		log.WithError(err).Error("error opening database")
+		return 1
+	}
+
+	if err = db.Merge(); err != nil {
 		log.WithError(err).Error("error merging database")
 		return 1
 	}
--- a/cmd/bitcask/root.go
+++ b/cmd/bitcask/root.go
@@ -8,13 +8,13 @@ import (
 	"github.com/spf13/cobra"
 	"github.com/spf13/viper"

-	"github.com/prologic/bitcask"
+	"github.com/prologic/bitcask/internal"
 )

 // RootCmd represents the base command when called without any subcommands
 var RootCmd = &cobra.Command{
 	Use:     "bitcask",
-	Version: bitcask.FullVersion(),
+	Version: internal.FullVersion(),
 	Short:   "Command-line tools for bitcask",
 	Long: `This is the command-line tool to interact with a bitcask database.

--- a/cmd/bitcask/scan.go
+++ b/cmd/bitcask/scan.go
@@ -0,0 +1,60 @@
+package main
+
+import (
+	"fmt"
+	"os"
+
+	log "github.com/sirupsen/logrus"
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+
+	"github.com/prologic/bitcask"
+)
+
+var scanCmd = &cobra.Command{
+	Use:     "scan <prefix>",
+	Aliases: []string{"search", "find"},
+	Short:   "Perform a prefis scan for keys",
+	Long: `This performa a prefix scan for keys  starting with the given
+prefix. This uses a Trie to search for matching keys and returns all matched
+keys.`,
+	Args: cobra.ExactArgs(1),
+	Run: func(cmd *cobra.Command, args []string) {
+		path := viper.GetString("path")
+
+		prefix := args[0]
+
+		os.Exit(scan(path, prefix))
+	},
+}
+
+func init() {
+	RootCmd.AddCommand(scanCmd)
+}
+
+func scan(path, prefix string) int {
+	db, err := bitcask.Open(path)
+	if err != nil {
+		log.WithError(err).Error("error opening database")
+		return 1
+	}
+	defer db.Close()
+
+	err = db.Scan(prefix, func(key string) error {
+		value, err := db.Get(key)
+		if err != nil {
+			log.WithError(err).Error("error reading key")
+			return err
+		}
+
+		fmt.Printf("%s\n", string(value))
+		log.WithField("key", key).WithField("value", value).Debug("key/value")
+		return nil
+	})
+	if err != nil {
+		log.WithError(err).Error("error scanning keys")
+		return 1
+	}
+
+	return 0
+}
--- a/cmd/bitcask/set.go
+++ b/cmd/bitcask/set.go
@@ -47,6 +47,7 @@ func set(path, key string, value io.Reader) int {
 		log.WithError(err).Error("error opening database")
 		return 1
 	}
+	defer db.Close()

 	data, err := ioutil.ReadAll(value)
 	if err != nil {
--- a/cmd/bitcask/stats.go
+++ b/cmd/bitcask/stats.go
@@ -0,0 +1,55 @@
+package main
+
+import (
+	"encoding/json"
+	"fmt"
+	"os"
+
+	log "github.com/sirupsen/logrus"
+	"github.com/spf13/cobra"
+	"github.com/spf13/viper"
+
+	"github.com/prologic/bitcask"
+)
+
+var statsCmd = &cobra.Command{
+	Use:     "stats",
+	Aliases: []string{},
+	Short:   "Display statis about the Database",
+	Long:    `This displays statistics about the Database"`,
+	Args:    cobra.ExactArgs(0),
+	Run: func(cmd *cobra.Command, args []string) {
+		path := viper.GetString("path")
+
+		os.Exit(stats(path))
+	},
+}
+
+func init() {
+	RootCmd.AddCommand(statsCmd)
+}
+
+func stats(path string) int {
+	db, err := bitcask.Open(path)
+	if err != nil {
+		log.WithError(err).Error("error opening database")
+		return 1
+	}
+	defer db.Close()
+
+	stats, err := db.Stats()
+	if err != nil {
+		log.WithError(err).Error("error getting stats")
+		return 1
+	}
+
+	data, err := json.MarshalIndent(stats, "", "  ")
+	if err != nil {
+		log.WithError(err).Error("error marshalling stats")
+		return 1
+	}
+
+	fmt.Println(string(data))
+
+	return 0
+}
--- a/cmd/bitcaskd/main.go
+++ b/cmd/bitcaskd/main.go
@@ -10,13 +10,14 @@ import (
 	"github.com/tidwall/redcon"

 	"github.com/prologic/bitcask"
+	"github.com/prologic/bitcask/internal"
 )

 var (
 	bind            string
 	debug           bool
 	version         bool
-	maxDatafileSize int64
+	maxDatafileSize int
 )

 func init() {
@@ -30,7 +31,7 @@ func init() {

 	flag.StringVarP(&bind, "bind", "b", ":6379", "interface and port to bind to")

-	flag.Int64Var(&maxDatafileSize, "max-datafile-size", 1<<20, "maximum datafile size in bytes")
+	flag.IntVar(&maxDatafileSize, "max-datafile-size", 1<<20, "maximum datafile size in bytes")
 }

 func main() {
@@ -43,7 +44,7 @@ func main() {
 	}

 	if version {
-		fmt.Printf("bitcaskd version %s", bitcask.FullVersion())
+		fmt.Printf("bitcaskd version %s", internal.FullVersion())
 		os.Exit(0)
 	}

@@ -60,7 +61,7 @@ func main() {
 		os.Exit(1)
 	}

-	log.WithField("bind", bind).WithField("path", path).Infof("starting bitcaskd v%s", bitcask.FullVersion())
+	log.WithField("bind", bind).WithField("path", path).Infof("starting bitcaskd v%s", internal.FullVersion())

 	err = redcon.ListenAndServe(bind,
 		func(conn redcon.Conn, cmd redcon.Command) {
@@ -95,6 +96,22 @@ func main() {
 				} else {
 					conn.WriteBulk(value)
 				}
+			case "keys":
+				conn.WriteArray(db.Len())
+				for key := range db.Keys() {
+					conn.WriteBulk([]byte(key))
+				}
+			case "exists":
+				if len(cmd.Args) != 2 {
+					conn.WriteError("ERR wrong number of arguments for '" + string(cmd.Args[0]) + "' command")
+					return
+				}
+				key := string(cmd.Args[1])
+				if db.Has(key) {
+					conn.WriteInt(1)
+				} else {
+					conn.WriteInt(0)
+				}
 			case "del":
 				if len(cmd.Args) != 2 {
 					conn.WriteError("ERR wrong number of arguments for '" + string(cmd.Args[0]) + "' command")
--- a/datafile.go
+++ b/datafile.go
@@ -1,151 +0,0 @@
-package bitcask
-
-import (
-	"errors"
-	"fmt"
-	"os"
-	"path/filepath"
-	"sync"
-	"time"
-
-	pb "github.com/prologic/bitcask/proto"
-	"github.com/prologic/bitcask/streampb"
-)
-
-const (
-	DefaultDatafileFilename = "%09d.data"
-)
-
-var (
-	ErrReadonly = errors.New("error: read only datafile")
-)
-
-type Datafile struct {
-	sync.RWMutex
-
-	id  int
-	r   *os.File
-	w   *os.File
-	dec *streampb.Decoder
-	enc *streampb.Encoder
-}
-
-func NewDatafile(path string, id int, readonly bool) (*Datafile, error) {
-	var (
-		r   *os.File
-		w   *os.File
-		err error
-	)
-
-	fn := filepath.Join(path, fmt.Sprintf(DefaultDatafileFilename, id))
-
-	if !readonly {
-		w, err = os.OpenFile(fn, os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0640)
-		if err != nil {
-			return nil, err
-		}
-	}
-
-	r, err = os.Open(fn)
-	if err != nil {
-		return nil, err
-	}
-
-	dec := streampb.NewDecoder(r)
-	enc := streampb.NewEncoder(w)
-
-	return &Datafile{
-		id:  id,
-		r:   r,
-		w:   w,
-		dec: dec,
-		enc: enc,
-	}, nil
-}
-
-func (df *Datafile) Name() string {
-	return df.r.Name()
-}
-
-func (df *Datafile) Close() error {
-	if df.w == nil {
-		return df.r.Close()
-	}
-
-	err := df.Sync()
-	if err != nil {
-		return err
-	}
-	return df.w.Close()
-}
-
-func (df *Datafile) Sync() error {
-	if df.w == nil {
-		return nil
-	}
-	return df.w.Sync()
-}
-
-func (df *Datafile) Size() (int64, error) {
-	var (
-		stat os.FileInfo
-		err  error
-	)
-
-	if df.w == nil {
-		stat, err = df.r.Stat()
-	} else {
-		stat, err = df.w.Stat()
-	}
-
-	if err != nil {
-		return -1, err
-	}
-
-	return stat.Size(), nil
-}
-
-func (df *Datafile) Read() (e pb.Entry, err error) {
-	df.Lock()
-	defer df.Unlock()
-
-	return e, df.dec.Decode(&e)
-}
-
-func (df *Datafile) ReadAt(index int64) (e pb.Entry, err error) {
-	df.Lock()
-	defer df.Unlock()
-
-	_, err = df.r.Seek(index, os.SEEK_SET)
-	if err != nil {
-		return
-	}
-
-	return e, df.dec.Decode(&e)
-}
-
-func (df *Datafile) Write(e pb.Entry) (int64, error) {
-	if df.w == nil {
-		return -1, ErrReadonly
-	}
-
-	stat, err := df.w.Stat()
-	if err != nil {
-		return -1, err
-	}
-
-	index := stat.Size()
-
-	e.Index = index
-	e.Timestamp = time.Now().Unix()
-
-	df.Lock()
-	err = df.enc.Encode(&e)
-	df.Unlock()
-
-	if err != nil {
-		return -1, err
-	}
-
-	return index, nil
-}
--- a/entry.go
+++ b/entry.go
@@ -1,17 +0,0 @@
-package bitcask
-
-import (
-	"hash/crc32"
-
-	pb "github.com/prologic/bitcask/proto"
-)
-
-func NewEntry(key string, value []byte) pb.Entry {
-	crc := crc32.ChecksumIEEE(value)
-
-	return pb.Entry{
-		CRC:   crc,
-		Key:   key,
-		Value: value,
-	}
-}
--- a/go.mod
+++ b/go.mod
@@ -3,18 +3,21 @@ module github.com/prologic/bitcask
 require (
 	github.com/gofrs/flock v0.7.1
 	github.com/gogo/protobuf v1.2.1
-	github.com/golang/protobuf v1.2.0
-	github.com/gorilla/websocket v1.4.0 // indirect
-	github.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7 // indirect
-	github.com/mitchellh/go-homedir v1.1.0
+	github.com/golang/protobuf v1.3.2
+	github.com/konsorten/go-windows-terminal-sequences v1.0.2 // indirect
+	github.com/magiconair/properties v1.8.1 // indirect
+	github.com/pelletier/go-toml v1.4.0 // indirect
 	github.com/pkg/errors v0.8.1
-	github.com/prologic/msgbus v0.1.1
-	github.com/prometheus/client_golang v0.9.2 // indirect
-	github.com/sirupsen/logrus v1.3.0
-	github.com/spf13/cobra v0.0.3
+	github.com/prologic/trie v0.0.0-20190322091023-3972df81f9b5
+	github.com/sirupsen/logrus v1.4.2
+	github.com/spf13/afero v1.2.2 // indirect
+	github.com/spf13/cobra v0.0.5
+	github.com/spf13/jwalterweatherman v1.1.0 // indirect
 	github.com/spf13/pflag v1.0.3
-	github.com/spf13/viper v1.3.1
+	github.com/spf13/viper v1.4.0
 	github.com/stretchr/testify v1.3.0
-	github.com/tidwall/redcon v0.9.0
-	gopkg.in/vmihailenco/msgpack.v2 v2.9.1
+	github.com/tidwall/redcon v1.0.0
+	golang.org/x/exp v0.0.0-20190718202018-cfdd5522f6f6
+	golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7 // indirect
+	golang.org/x/text v0.3.2 // indirect
 )
--- a/go.sum
+++ b/go.sum
@@ -1,89 +1,198 @@
+cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
+github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
+github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
+github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
+github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
+github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
+github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
 github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
-github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973 h1:xJ4a3vCFaGF/jqvzLMYoU8P317H5OQ+Via4RmuPwCS0=
 github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
+github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
+github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
+github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
+github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
 github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
 github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
 github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
+github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
+github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
+github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=
 github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
+github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
 github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
+github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
+github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
+github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
+github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
+github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
 github.com/gofrs/flock v0.7.1 h1:DP+LD/t0njgoPBvT5MJLeliUIVQR03hiKR6vezdwHlc=
 github.com/gofrs/flock v0.7.1/go.mod h1:F1TvTiK9OcQqauNUHlbJvyl9Qa1QvF/gOUDKA14jxHU=
+github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
 github.com/gogo/protobuf v1.2.1 h1:/s5zKNz0uPFCZ5hddgPdo2TK2TVrUNMn0OOX8/aZMTE=
 github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
-github.com/golang/protobuf v1.2.0 h1:P3YflyNX/ehuJFLhxviNdFxQPkGK5cDcApsge1SqnvM=
+github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
+github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
-github.com/gorilla/websocket v1.4.0 h1:WDFjx/TMzVgy9VdMMQi2K2Emtwi2QcUQsztZ/zLaH/Q=
+github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg=
+github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs=
+github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
+github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
+github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
 github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
+github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
+github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
+github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
 github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
 github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
-github.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7 h1:K//n/AqR5HjG3qxbrBCL4vJPW0MVFSs9CPK1OOJdRME=
-github.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7/go.mod h1:2iMrUgbbvHEiQClaW2NsSzMyGHqN+rDFqY705q49KG0=
+github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
+github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
+github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
+github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
 github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
 github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
+github.com/konsorten/go-windows-terminal-sequences v1.0.1 h1:mweAR1A6xJ3oS2pRaGiHgQ4OO8tzTaLawm8vnODuwDk=
 github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
+github.com/konsorten/go-windows-terminal-sequences v1.0.2 h1:DB17ag19krx9CFsz4o3enTrPXyIXCl+2iCXH/aMAp9s=
+github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
+github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
+github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/magiconair/properties v1.8.0 h1:LLgXmsheXeRoUOBOjtwPQCWIYqM/LU1ayDtDePerRcY=
 github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
-github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU=
+github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=
+github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
-github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
 github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
 github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE=
 github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
+github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
+github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
 github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
 github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
+github.com/pelletier/go-toml v1.4.0 h1:u3Z1r+oOXJIkxqw34zVhyPgjBsm6X2wn21NWs/HfSeg=
+github.com/pelletier/go-toml v1.4.0/go.mod h1:PN7xzY2wHTK0K9p34ErDQMlFxa51Fk0OUruD3k1mMwo=
+github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.8.1 h1:iURUrRGxPUNPdy5/HRSm+Yj6okJ6UtLINN0Q9M4+h3I=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/prologic/msgbus v0.1.1/go.mod h1:B3Qu4/U2FP08x93jUzp9E8bl155+cIgDH2DUGRK6OZk=
-github.com/prometheus/client_golang v0.9.2 h1:awm861/B8OKDd2I/6o1dy3ra4BamzKhYOiGItCeZ740=
-github.com/prometheus/client_golang v0.9.2/go.mod h1:OsXs2jCmiKlQ1lTBmv21f2mNfw4xf/QclQDMrYNZzcM=
-github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910 h1:idejC8f05m9MGOsuEi1ATq9shN03HrxNkD/luQvxCv8=
+github.com/prologic/trie v0.0.0-20190322091023-3972df81f9b5 h1:H8dTZzU3aWNQnuRyiT45J9szv7EFakAhFzsFq27t3Uo=
+github.com/prologic/trie v0.0.0-20190322091023-3972df81f9b5/go.mod h1:LFuDmpHJGmciXd8Rl5YMhVlLMps9gz2GtYLzwxrFhzs=
+github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
+github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
 github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
-github.com/prometheus/common v0.0.0-20181126121408-4724e9255275 h1:PnBWHBf+6L0jOqq0gIVUe6Yk0/QMZ640k6NvkxcBf+8=
-github.com/prometheus/common v0.0.0-20181126121408-4724e9255275/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
-github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a h1:9a8MnZMP0X2nLJdBg+pBmGgkJlSaKC2KaQmTCk1XDtE=
-github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
-github.com/sirupsen/logrus v1.3.0 h1:hI/7Q+DtNZ2kINb6qt/lS+IyXnHQe9e90POfeewL/ME=
-github.com/sirupsen/logrus v1.3.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
+github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
+github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
+github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
+github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
+github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
+github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
+github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
+github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
+github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
+github.com/sirupsen/logrus v1.4.2 h1:SPIRibHv4MatM3XXNO2BJeFLZwZ2LvZgfQ5+UNI2im4=
+github.com/sirupsen/logrus v1.4.2/go.mod h1:tLMulIdttU9McNUspp0xgXVQah82FyeX6MwdIuYE2rE=
+github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
+github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
 github.com/spf13/afero v1.1.2 h1:m8/z1t7/fwjysjQRYbP0RD+bUIF/8tJwPdEZsI83ACI=
 github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
+github.com/spf13/afero v1.2.2 h1:5jhuqJyZCZf2JRofRvN/nIFgIWNzPa3/Vz8mYylgbWc=
+github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
 github.com/spf13/cast v1.3.0 h1:oget//CVOEoFewqQxwr0Ej5yjygnqGkvggSE/gB35Q8=
 github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
-github.com/spf13/cobra v0.0.3 h1:ZlrZ4XsMRm04Fr5pSFxBgfND2EBVa1nLpiy1stUsX/8=
-github.com/spf13/cobra v0.0.3/go.mod h1:1l0Ry5zgKvJasoi3XT1TypsSe7PqH0Sj9dhYf7v3XqQ=
+github.com/spf13/cobra v0.0.5 h1:f0B+LkLX6DtmRH1isoNA9VTtNUK9K8xYd28JNNfOv/s=
+github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
 github.com/spf13/jwalterweatherman v1.0.0 h1:XHEdyB+EcvlqZamSM4ZOMGlc93t6AcsBEu9Gc1vn7yk=
 github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
+github.com/spf13/jwalterweatherman v1.1.0 h1:ue6voC5bR5F8YxI5S67j9i582FU4Qvo2bmqnqMYADFk=
+github.com/spf13/jwalterweatherman v1.1.0/go.mod h1:aNWZUN0dPAAO/Ljvb5BEdw96iTZ0EXowPYD95IqWIGo=
 github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
 github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
-github.com/spf13/viper v1.3.1 h1:5+8j8FTpnFV4nEImW/ofkzEt8VoOiLXxdYIDsB73T38=
-github.com/spf13/viper v1.3.1/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
+github.com/spf13/viper v1.3.2 h1:VUFqw5KcqRf7i70GOzW7N+Q7+gxVBkSSqiXB12+JQ4M=
+github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
+github.com/spf13/viper v1.4.0 h1:yXHLWeravcrgGyFSyCgdYpXQ9dR9c/WED3pg1RhxqEU=
+github.com/spf13/viper v1.4.0/go.mod h1:PTJ7Z/lr49W6bUbkmS1V3by4uWynFiR9p7+dSq/yZzE=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
 github.com/stretchr/testify v1.3.0 h1:TivCn/peBQ7UY8ooIcPgZFpTNSz0Q2U6UrFlUfqbe0Q=
 github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
-github.com/tidwall/redcon v0.9.0 h1:tiT9DLAoohsdNaFg9Si5dRsv9+FjvZYnhMOEtSFwBqA=
-github.com/tidwall/redcon v0.9.0/go.mod h1:bdYBm4rlcWpst2XMwKVzWDF9CoUxEbUmM7CQrKeOZas=
+github.com/tidwall/redcon v1.0.0 h1:D4AzzJ81Afeh144fgnj5H0aSVPBBJ5RI9Rzj0zThU+E=
+github.com/tidwall/redcon v1.0.0/go.mod h1:bdYBm4rlcWpst2XMwKVzWDF9CoUxEbUmM7CQrKeOZas=
+github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
+github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
 github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
+github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
 github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
+go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
+go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
+go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
+go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
 golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9 h1:mKdxBk7AujPs8kU4m80U72y/zjbZ3UcXC7dClwKbUI0=
 golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
-golang.org/x/net v0.0.0-20181201002055-351d144fa1fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/exp v0.0.0-20190718202018-cfdd5522f6f6 h1:1xlTfQXUyQvvbx4TFsXRgoj9r0YUwcqq9XGX9u9OODU=
+golang.org/x/exp v0.0.0-20190718202018-cfdd5522f6f6/go.mod h1:JhuoJpWY28nO4Vef9tZUw9qufEGTyX1+7lmHxV5q5G4=
+golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
+golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
+golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
+golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
+golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
+golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
+golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a h1:1n5lsVfiQW3yfsRGu98756EH1YthsFqr/5mxHduZW2A=
 golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7 h1:LepdCS8Gf/MVejFIt8lsiexZATdoGVyp5bcyS+rYoUI=
+golang.org/x/sys v0.0.0-20190712062909-fae7ac547cb7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
+golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
+golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
+golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
+google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
+google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
+google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
+gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
-gopkg.in/vmihailenco/msgpack.v2 v2.9.1 h1:kb0VV7NuIojvRfzwslQeP3yArBqJHW9tOl4t38VS1jM=
-gopkg.in/vmihailenco/msgpack.v2 v2.9.1/go.mod h1:/3Dn1Npt9+MYyLpYYXjInO/5jvMLamn+AEGwNEOatn8=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
+gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
+gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
--- a/internal/datafile.go
+++ b/internal/datafile.go
@@ -0,0 +1,177 @@
+package internal
+
+import (
+	"bytes"
+	"fmt"
+	"os"
+	"path/filepath"
+	"sync"
+
+	"github.com/pkg/errors"
+	"golang.org/x/exp/mmap"
+
+	pb "github.com/prologic/bitcask/internal/proto"
+	"github.com/prologic/bitcask/internal/streampb"
+)
+
+const (
+	DefaultDatafileFilename = "%09d.data"
+)
+
+var (
+	ErrReadonly  = errors.New("error: read only datafile")
+	ErrReadError = errors.New("error: read error")
+)
+
+type Datafile struct {
+	sync.RWMutex
+
+	id     int
+	r      *os.File
+	ra     *mmap.ReaderAt
+	w      *os.File
+	offset int64
+	dec    *streampb.Decoder
+	enc    *streampb.Encoder
+}
+
+func NewDatafile(path string, id int, readonly bool) (*Datafile, error) {
+	var (
+		r   *os.File
+		ra  *mmap.ReaderAt
+		w   *os.File
+		err error
+	)
+
+	fn := filepath.Join(path, fmt.Sprintf(DefaultDatafileFilename, id))
+
+	if !readonly {
+		w, err = os.OpenFile(fn, os.O_WRONLY|os.O_APPEND|os.O_CREATE, 0640)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	r, err = os.Open(fn)
+	if err != nil {
+		return nil, err
+	}
+	stat, err := r.Stat()
+	if err != nil {
+		return nil, errors.Wrap(err, "error calling Stat()")
+	}
+
+	ra, err = mmap.Open(fn)
+	if err != nil {
+		return nil, err
+	}
+
+	offset := stat.Size()
+
+	dec := streampb.NewDecoder(r)
+	enc := streampb.NewEncoder(w)
+
+	return &Datafile{
+		id:     id,
+		r:      r,
+		ra:     ra,
+		w:      w,
+		offset: offset,
+		dec:    dec,
+		enc:    enc,
+	}, nil
+}
+
+func (df *Datafile) FileID() int {
+	return df.id
+}
+
+func (df *Datafile) Name() string {
+	return df.r.Name()
+}
+
+func (df *Datafile) Close() error {
+	defer func() {
+		df.ra.Close()
+		df.r.Close()
+	}()
+
+	// Readonly Datafile -- Nothing further to close on the write side
+	if df.w == nil {
+		return nil
+	}
+
+	err := df.Sync()
+	if err != nil {
+		return err
+	}
+	return df.w.Close()
+}
+
+func (df *Datafile) Sync() error {
+	if df.w == nil {
+		return nil
+	}
+	return df.w.Sync()
+}
+
+func (df *Datafile) Size() int64 {
+	df.RLock()
+	defer df.RUnlock()
+	return df.offset
+}
+
+func (df *Datafile) Read() (e pb.Entry, n int64, err error) {
+	df.Lock()
+	defer df.Unlock()
+
+	n, err = df.dec.Decode(&e)
+	if err != nil {
+		return
+	}
+
+	return
+}
+
+func (df *Datafile) ReadAt(index, size int64) (e pb.Entry, err error) {
+	var n int
+
+	b := make([]byte, size)
+
+	if df.w == nil {
+		n, err = df.ra.ReadAt(b, index)
+	} else {
+		n, err = df.r.ReadAt(b, index)
+	}
+	if err != nil {
+		return
+	}
+	if int64(n) != size {
+		err = ErrReadError
+		return
+	}
+
+	buf := bytes.NewBuffer(b)
+	dec := streampb.NewDecoder(buf)
+	_, err = dec.Decode(&e)
+	return
+}
+
+func (df *Datafile) Write(e pb.Entry) (int64, int64, error) {
+	if df.w == nil {
+		return -1, 0, ErrReadonly
+	}
+
+	df.Lock()
+	defer df.Unlock()
+
+	e.Offset = df.offset
+
+	n, err := df.enc.Encode(&e)
+	if err != nil {
+		return -1, 0, err
+	}
+	df.offset += n
+
+	return e.Offset, n, nil
+}
--- a/internal/entry.go
+++ b/internal/entry.go
@@ -0,0 +1,17 @@
+package internal
+
+import (
+	"hash/crc32"
+
+	pb "github.com/prologic/bitcask/internal/proto"
+)
+
+func NewEntry(key string, value []byte) pb.Entry {
+	checksum := crc32.ChecksumIEEE(value)
+
+	return pb.Entry{
+		Checksum: checksum,
+		Key:      key,
+		Value:    value,
+	}
+}
--- a/internal/keydir.go
+++ b/internal/keydir.go
@@ -1,17 +1,18 @@
-package bitcask
+package internal

 import (
 	"bytes"
 	"encoding/gob"
 	"io"
 	"io/ioutil"
+	"os"
 	"sync"
 )

 type Item struct {
-	FileID    int
-	Index     int64
-	Timestamp int64
+	FileID int
+	Offset int64
+	Size   int64
 }

 type Keydir struct {
@@ -25,15 +26,18 @@ func NewKeydir() *Keydir {
 	}
 }

-func (k *Keydir) Add(key string, fileid int, index, timestamp int64) {
-	k.Lock()
-	defer k.Unlock()
-
-	k.kv[key] = Item{
-		FileID:    fileid,
-		Index:     index,
-		Timestamp: timestamp,
+func (k *Keydir) Add(key string, fileid int, offset, size int64) Item {
+	item := Item{
+		FileID: fileid,
+		Offset: offset,
+		Size:   size,
 	}
+
+	k.Lock()
+	k.kv[key] = item
+	k.Unlock()
+
+	return item
 }

 func (k *Keydir) Get(key string) (Item, bool) {
@@ -51,11 +55,17 @@ func (k *Keydir) Delete(key string) {
 	delete(k.kv, key)
 }

+func (k *Keydir) Len() int {
+	return len(k.kv)
+}
+
 func (k *Keydir) Keys() chan string {
 	ch := make(chan string)
 	go func() {
-		for k := range k.kv {
-			ch <- k
+		k.RLock()
+		defer k.RUnlock()
+		for key := range k.kv {
+			ch <- key
 		}
 		close(ch)
 	}()
@@ -72,6 +82,21 @@ func (k *Keydir) Bytes() ([]byte, error) {
 	return buf.Bytes(), nil
 }

+func (k *Keydir) Load(fn string) error {
+	f, err := os.Open(fn)
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+
+	dec := gob.NewDecoder(f)
+	if err := dec.Decode(&k.kv); err != nil {
+		return err
+	}
+
+	return nil
+}
+
 func (k *Keydir) Save(fn string) error {
 	data, err := k.Bytes()
 	if err != nil {
--- a/internal/proto/doc.go
+++ b/internal/proto/doc.go
--- a/internal/proto/entry.pb.go
+++ b/internal/proto/entry.pb.go
@@ -19,11 +19,10 @@ var _ = math.Inf
 const _ = proto.ProtoPackageIsVersion2 // please upgrade the proto package

 type Entry struct {
-	CRC                  uint32   `protobuf:"varint,1,opt,name=CRC,proto3" json:"CRC,omitempty"`
+	Checksum             uint32   `protobuf:"varint,1,opt,name=Checksum,proto3" json:"Checksum,omitempty"`
 	Key                  string   `protobuf:"bytes,2,opt,name=Key,proto3" json:"Key,omitempty"`
-	Index                int64    `protobuf:"varint,3,opt,name=Index,proto3" json:"Index,omitempty"`
+	Offset               int64    `protobuf:"varint,3,opt,name=Offset,proto3" json:"Offset,omitempty"`
 	Value                []byte   `protobuf:"bytes,4,opt,name=Value,proto3" json:"Value,omitempty"`
-	Timestamp            int64    `protobuf:"varint,5,opt,name=Timestamp,proto3" json:"Timestamp,omitempty"`
 	XXX_NoUnkeyedLiteral struct{} `json:"-"`
 	XXX_unrecognized     []byte   `json:"-"`
 	XXX_sizecache        int32    `json:"-"`
@@ -33,7 +32,7 @@ func (m *Entry) Reset()         { *m = Entry{} }
 func (m *Entry) String() string { return proto.CompactTextString(m) }
 func (*Entry) ProtoMessage()    {}
 func (*Entry) Descriptor() ([]byte, []int) {
-	return fileDescriptor_entry_4f5906245d08394f, []int{0}
+	return fileDescriptor_entry_db5b99f271e6b4b6, []int{0}
 }
 func (m *Entry) XXX_Unmarshal(b []byte) error {
 	return xxx_messageInfo_Entry.Unmarshal(m, b)
@@ -53,9 +52,9 @@ func (m *Entry) XXX_DiscardUnknown() {

 var xxx_messageInfo_Entry proto.InternalMessageInfo

-func (m *Entry) GetCRC() uint32 {
+func (m *Entry) GetChecksum() uint32 {
 	if m != nil {
-		return m.CRC
+		return m.Checksum
 	}
 	return 0
 }
@@ -67,9 +66,9 @@ func (m *Entry) GetKey() string {
 	return ""
 }

-func (m *Entry) GetIndex() int64 {
+func (m *Entry) GetOffset() int64 {
 	if m != nil {
-		return m.Index
+		return m.Offset
 	}
 	return 0
 }
@@ -81,28 +80,20 @@ func (m *Entry) GetValue() []byte {
 	return nil
 }

-func (m *Entry) GetTimestamp() int64 {
-	if m != nil {
-		return m.Timestamp
-	}
-	return 0
-}
-
 func init() {
 	proto.RegisterType((*Entry)(nil), "proto.Entry")
 }

-func init() { proto.RegisterFile("entry.proto", fileDescriptor_entry_4f5906245d08394f) }
+func init() { proto.RegisterFile("entry.proto", fileDescriptor_entry_db5b99f271e6b4b6) }

-var fileDescriptor_entry_4f5906245d08394f = []byte{
-	// 134 bytes of a gzipped FileDescriptorProto
+var fileDescriptor_entry_db5b99f271e6b4b6 = []byte{
+	// 126 bytes of a gzipped FileDescriptorProto
 	0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0xe2, 0x4e, 0xcd, 0x2b, 0x29,
-	0xaa, 0xd4, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0x62, 0x05, 0x53, 0x4a, 0xa5, 0x5c, 0xac, 0xae,
-	0x20, 0x51, 0x21, 0x01, 0x2e, 0x66, 0xe7, 0x20, 0x67, 0x09, 0x46, 0x05, 0x46, 0x0d, 0xde, 0x20,
-	0x10, 0x13, 0x24, 0xe2, 0x9d, 0x5a, 0x29, 0xc1, 0xa4, 0xc0, 0xa8, 0xc1, 0x19, 0x04, 0x62, 0x0a,
-	0x89, 0x70, 0xb1, 0x7a, 0xe6, 0xa5, 0xa4, 0x56, 0x48, 0x30, 0x2b, 0x30, 0x6a, 0x30, 0x07, 0x41,
-	0x38, 0x20, 0xd1, 0xb0, 0xc4, 0x9c, 0xd2, 0x54, 0x09, 0x16, 0x05, 0x46, 0x0d, 0x9e, 0x20, 0x08,
-	0x47, 0x48, 0x86, 0x8b, 0x33, 0x24, 0x33, 0x37, 0xb5, 0xb8, 0x24, 0x31, 0xb7, 0x40, 0x82, 0x15,
-	0xac, 0x1e, 0x21, 0x90, 0xc4, 0x06, 0xb6, 0xdd, 0x18, 0x10, 0x00, 0x00, 0xff, 0xff, 0x07, 0x99,
-	0x47, 0xb9, 0x93, 0x00, 0x00, 0x00,
+	0xaa, 0xd4, 0x2b, 0x28, 0xca, 0x2f, 0xc9, 0x17, 0x62, 0x05, 0x53, 0x4a, 0xc9, 0x5c, 0xac, 0xae,
+	0x20, 0x51, 0x21, 0x29, 0x2e, 0x0e, 0xe7, 0x8c, 0xd4, 0xe4, 0xec, 0xe2, 0xd2, 0x5c, 0x09, 0x46,
+	0x05, 0x46, 0x0d, 0xde, 0x20, 0x38, 0x5f, 0x48, 0x80, 0x8b, 0xd9, 0x3b, 0xb5, 0x52, 0x82, 0x49,
+	0x81, 0x51, 0x83, 0x33, 0x08, 0xc4, 0x14, 0x12, 0xe3, 0x62, 0xf3, 0x4f, 0x4b, 0x2b, 0x4e, 0x2d,
+	0x91, 0x60, 0x56, 0x60, 0xd4, 0x60, 0x0e, 0x82, 0xf2, 0x84, 0x44, 0xb8, 0x58, 0xc3, 0x12, 0x73,
+	0x4a, 0x53, 0x25, 0x58, 0x14, 0x18, 0x35, 0x78, 0x82, 0x20, 0x9c, 0x24, 0x36, 0xb0, 0x5d, 0xc6,
+	0x80, 0x00, 0x00, 0x00, 0xff, 0xff, 0x76, 0xd2, 0x3e, 0x83, 0x81, 0x00, 0x00, 0x00,
 }
--- a/internal/proto/entry.proto
+++ b/internal/proto/entry.proto
@@ -3,9 +3,8 @@ syntax = "proto3";
 package proto;

 message Entry {
-	uint32 CRC = 1;
+	uint32 Checksum = 1;
 	string Key = 2;
-	int64 Index = 3;
+	int64 Offset = 3;
 	bytes Value = 4;
-	int64 Timestamp = 5;
 }
--- a/internal/streampb/stream.go
+++ b/internal/streampb/stream.go
@@ -1,6 +1,7 @@
 package streampb

 import (
+	"bufio"
 	"encoding/binary"
 	"io"

@@ -16,32 +17,40 @@ const (

 // NewEncoder creates a streaming protobuf encoder.
 func NewEncoder(w io.Writer) *Encoder {
-	return &Encoder{w}
+	return &Encoder{w: bufio.NewWriter(w)}
 }

 // Encoder wraps an underlying io.Writer and allows you to stream
 // proto encodings on it.
 type Encoder struct {
-	w io.Writer
+	w *bufio.Writer
 }

 // Encode takes any proto.Message and streams it to the underlying writer.
 // Messages are framed with a length prefix.
-func (e *Encoder) Encode(msg proto.Message) error {
+func (e *Encoder) Encode(msg proto.Message) (int64, error) {
 	prefixBuf := make([]byte, prefixSize)

 	buf, err := proto.Marshal(msg)
 	if err != nil {
-		return err
+		return 0, err
 	}
 	binary.BigEndian.PutUint64(prefixBuf, uint64(len(buf)))

 	if _, err := e.w.Write(prefixBuf); err != nil {
-		return errors.Wrap(err, "failed writing length prefix")
+		return 0, errors.Wrap(err, "failed writing length prefix")
 	}

-	_, err = e.w.Write(buf)
-	return errors.Wrap(err, "failed writing marshaled data")
+	n, err := e.w.Write(buf)
+	if err != nil {
+		return 0, errors.Wrap(err, "failed writing marshaled data")
+	}
+
+	if err = e.w.Flush(); err != nil {
+		return 0, errors.Wrap(err, "failed flushing data")
+	}
+
+	return int64(n + prefixSize), nil
 }

 // NewDecoder creates a streaming protobuf decoder.
@@ -57,12 +66,12 @@ type Decoder struct {

 // Decode takes a proto.Message and unmarshals the next payload in the
 // underlying io.Reader. It returns an EOF when it's done.
-func (d *Decoder) Decode(v proto.Message) error {
+func (d *Decoder) Decode(v proto.Message) (int64, error) {
 	prefixBuf := make([]byte, prefixSize)

 	_, err := io.ReadFull(d.r, prefixBuf)
 	if err != nil {
-		return err
+		return 0, err
 	}

 	n := binary.BigEndian.Uint64(prefixBuf)
@@ -73,11 +82,11 @@ func (d *Decoder) Decode(v proto.Message) error {
 	for idx < n {
 		m, err := d.r.Read(buf[idx:n])
 		if err != nil {
-			return errors.Wrap(translateError(err), "failed reading marshaled data")
+			return 0, errors.Wrap(translateError(err), "failed reading marshaled data")
 		}
 		idx += uint64(m)
 	}
-	return proto.Unmarshal(buf[:n], v)
+	return int64(idx + prefixSize), proto.Unmarshal(buf[:n], v)
 }

 func translateError(err error) error {
--- a/internal/utils.go
+++ b/internal/utils.go
@@ -0,0 +1,56 @@
+package internal
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
+	"strconv"
+	"strings"
+)
+
+func Exists(path string) bool {
+	_, err := os.Stat(path)
+	return err == nil
+}
+
+func DirSize(path string) (int64, error) {
+	var size int64
+	err := filepath.Walk(path, func(_ string, info os.FileInfo, err error) error {
+		if err != nil {
+			return err
+		}
+		if !info.IsDir() {
+			size += info.Size()
+		}
+		return err
+	})
+	return size, err
+}
+
+func GetDatafiles(path string) ([]string, error) {
+	fns, err := filepath.Glob(fmt.Sprintf("%s/*.data", path))
+	if err != nil {
+		return nil, err
+	}
+	sort.Strings(fns)
+	return fns, nil
+}
+
+func ParseIds(fns []string) ([]int, error) {
+	var ids []int
+	for _, fn := range fns {
+		fn = filepath.Base(fn)
+		ext := filepath.Ext(fn)
+		if ext != ".data" {
+			continue
+		}
+		id, err := strconv.ParseInt(strings.TrimSuffix(fn, ext), 10, 32)
+		if err != nil {
+			return nil, err
+		}
+		ids = append(ids, int(id))
+	}
+	sort.Ints(ids)
+	return ids, nil
+}
--- a/internal/version.go
+++ b/internal/version.go
@@ -1,4 +1,4 @@
-package bitcask
+package internal

 import (
 	"fmt"
--- a/internal/version_test.go
+++ b/internal/version_test.go
@@ -1,4 +1,4 @@
-package bitcask
+package internal

 import (
 	"fmt"
--- a/options.go
+++ b/options.go
@@ -0,0 +1,53 @@
+package bitcask
+
+const (
+	// DefaultMaxDatafileSize is the default maximum datafile size in bytes
+	DefaultMaxDatafileSize = 1 << 20 // 1MB
+
+	// DefaultMaxKeySize is the default maximum key size in bytes
+	DefaultMaxKeySize = 64 // 64 bytes
+
+	// DefaultMaxValueSize is the default value size in bytes
+	DefaultMaxValueSize = 1 << 16 // 65KB
+)
+
+// Option is a function that takes a config struct and modifies it
+type Option func(*config) error
+
+type config struct {
+	maxDatafileSize int
+	maxKeySize      int
+	maxValueSize    int
+}
+
+func newDefaultConfig() *config {
+	return &config{
+		maxDatafileSize: DefaultMaxDatafileSize,
+		maxKeySize:      DefaultMaxKeySize,
+		maxValueSize:    DefaultMaxValueSize,
+	}
+}
+
+// WithMaxDatafileSize sets the maximum datafile size option
+func WithMaxDatafileSize(size int) Option {
+	return func(cfg *config) error {
+		cfg.maxDatafileSize = size
+		return nil
+	}
+}
+
+// WithMaxKeySize sets the maximum key size option
+func WithMaxKeySize(size int) Option {
+	return func(cfg *config) error {
+		cfg.maxKeySize = size
+		return nil
+	}
+}
+
+// WithMaxValueSize sets the maximum value size option
+func WithMaxValueSize(size int) Option {
+	return func(cfg *config) error {
+		cfg.maxValueSize = size
+		return nil
+	}
+}
--- a/tools/release.sh
+++ b/tools/release.sh
@@ -20,6 +20,6 @@ fi

 echo "Releasing ${TAG} ..."

-git tag -a -s -m "Relase ${TAG}" "${TAG}"
+git tag -a -s -m "Release ${TAG}" "${TAG}"
 git push --tags
 goreleaser release --rm-dist
Author	SHA1	Message	Date
James Mills	912371645d	Fixed an off-by-one bug with managing datafiles (#31 )	2019-07-29 23:49:37 +10:00
James Mills	bc782a3083	Update README.md	2019-07-27 07:57:38 +10:00
James Mills	a2161179ef	Update README.md	2019-07-27 07:56:34 +10:00
James Mills	51bac21c0a	Improves Merge() operation by also pruning old key/value pairs (#29 ) * Added new API Stats() and Prune() * Improved Merge() logic to also prune old key/values and actually reclaim disk space * Added backward compat for the old Merge() function * Refactor indexing of keys to items (hints) * Remove redundant TestOpenMerge * Add unit test for Stats() * Improve TestMerge()	2019-07-27 07:52:25 +10:00
Awn	b7ac95d66a	remove merge folder after merge completes (#26 )	2019-07-25 08:31:44 +10:00
Awn	c28c72108f	Use consistent directory names in benchmarks (#24 ) * patch: use current directory for benchmarks * Use consistent directory names	2019-07-25 08:18:41 +10:00
Awn	a74203b99e	patch: use current directory for benchmarks (#23 )	2019-07-25 08:09:04 +10:00
Awn	5ee0f8e0df	update dependencies (#22 )	2019-07-25 08:02:00 +10:00
Awn	479cabcc8e	patch: use directory within database path for merge (#21 )	2019-07-25 08:01:12 +10:00
Awn	3b63388e79	benchmarks: set test size to compute throughput (#16 ) * benchmarks: set test size to compute throughput * patch: error on cross-filesystem rename operation * revert: commit to wrong branch	2019-07-25 07:14:58 +10:00
panyun	fd2023ee38	Fix issue(db file Merge issue in windows env): (#15 ) * Fix issue(windows env): 1. Run my program firstly, it will create my.db folder normaly. 2. Close my program and restart it again, I will receive an error. * Fix coding format issue * Change the last fixing with @prologic suggestion.	2019-07-23 13:27:14 +10:00
James Mills	47ad6601f3	Update generated protobuf code	2019-07-19 20:49:49 +10:00
Christian Muehlhaeuser	057c147f89	Added missing error check (#13 ) err was assigned, but not checked.	2019-07-19 20:22:56 +10:00
Jesse Donat	9fafcad9a6	Fix mismatched key casing. (#12 )	2019-06-12 07:29:11 +10:00
Whemoon Jang	c4faac9f7c	Fix outdated README (#11 )	2019-06-03 20:48:30 +10:00
Yury Fedorov	43334647a6	Fix typos in bitcask.go docs (#10 )	2019-05-26 19:27:03 +10:00
Christian Muehlhaeuser	f26a1b1727	Fixed typo in release.sh (#8 )	2019-04-25 22:19:20 +10:00
Kebert Xela	1fca55d268	Minor readme improvements (#7 ) * File emoji and missing parenthesis * Update README.md * Update README.md	2019-04-24 06:44:21 +10:00
James Mills	c640f7f7e7	Update README.md	2019-04-01 17:36:18 +10:00
James Mills	b6c9867e7b	Added missing dependency on github.com/golang/protobuf/proto Fixes #6	2019-04-01 17:29:23 +10:00
James Mills	ed6283dca4	Add Development section to README documenting use of Protobuf and tooling required. #6	2019-04-01 17:29:23 +10:00
James Mills	f44b6249ac	Create CONTRIBUTING.md	2019-04-01 10:14:40 +10:00
James Mills	2be3a63153	Add other badges from img.shields.io	2019-03-31 13:39:58 +10:00
James Mills	836deeb0ff	Document using the Docker Image	2019-03-30 20:25:23 +10:00
James Mills	b29b4c5422	Add Dockerfile to publish images to Docker Hub	2019-03-30 17:14:09 +10:00
James Mills	f397a73abd	Update README.md	2019-03-29 22:21:39 +10:00
James Mills	53dc013215	Optimized and increased read performance by ~2-3x by memory mapping the read-only datafiles	2019-03-23 13:35:55 +10:00
James Mills	711d08ce91	Increased read performance by ~3-4x by removing another unnecessary I/O operation (Seek)	2019-03-23 12:14:15 +10:00
James Mills	c3b1a02371	Updated deps	2019-03-22 21:53:09 +10:00
James Mills	7149cb9afe	Fixed concurrency bug with reopening datafiles when maxDatafileSize is exceeded	2019-03-22 17:59:07 +10:00
James Mills	c593bc966f	Refactor Datafile.Size()	2019-03-22 17:33:24 +10:00
James Mills	2400dd86d5	Add docs for bitcask	2019-03-21 17:46:53 +10:00
James Mills	27eb922ba2	Add docs for options	2019-03-21 17:20:53 +10:00
James Mills	34ad78efc0	Add KeYS command to server (bitraftd)	2019-03-21 10:49:53 +10:00
James Mills	352c32ee12	Add Len() to exported API (extended API)	2019-03-21 10:47:50 +10:00
James Mills	aaea7273c3	Add Keys() to exported API (extended API)	2019-03-21 10:41:56 +10:00
James Mills	01cb269a51	Add EXISTS command to server (bitraftd)	2019-03-21 10:29:18 +10:00
James Mills	962e53af17	Add Has() to exported API (extended API)	2019-03-21 10:24:48 +10:00
James Mills	7a427a237a	Update README.md	2019-03-21 08:36:17 +10:00
James Mills	8bf169c96f	Add MergeOpen test case	2019-03-20 17:10:24 +10:00
James Mills	c1488fed2a	Added Fold() test case	2019-03-20 16:55:59 +10:00
James Mills	d6e806e655	Update README.md	2019-03-20 15:30:08 +10:00
James Mills	2d9bfbb408	Unexport NewDefaultConfig (not useful for public consumption)	2019-03-20 07:46:26 +10:00
James Mills	d8a48f9eea	Use pre-defined errors as they are comparable and useful as exported symbols	2019-03-20 07:39:03 +10:00
James Mills	65e7877bdf	Remove notify step for now	2019-03-20 07:08:10 +10:00
James Mills	5711478dd6	Drone CI plugins now use a new key called settings :/	2019-03-20 07:06:31 +10:00
James Mills	336795285e	Fixed Drone CI config	2019-03-20 07:02:39 +10:00
James Mills	7fba9bd4b7	Add bitcaskd to install target	2019-03-20 07:00:46 +10:00
James Mills	e117ffd2e9	Fixed injecting Version/Commit in relased binaries	2019-03-19 18:55:03 +10:00
James Mills	ebefd0abf4	Updated deps	2019-03-18 19:47:08 +10:00
James Mills	52dfec6760	Only count coverage for the exported package	2019-03-18 17:40:45 +10:00
James Mills	1298240f53	Unexport some internal implemtnation details	2019-03-18 17:31:31 +10:00
James Mills	2a35976cdd	Ooops	2019-03-17 14:00:15 +10:00
James Mills	6fe6fe0689	Refactored configuration option handling. Fixes #3	2019-03-17 13:53:30 +10:00
James Mills	e83608b903	Fixed missing error handling opening new Datafile(s) during Put() Fixes #4	2019-03-17 13:47:07 +10:00
James Mills	67ab944db7	Refactored some internals and removed timestamp field (unsure why it was needed in the original paper)	2019-03-16 12:40:24 +10:00
James Mills	cb00b11dd7	Increase no. of goroutines to catch more race conditions in tests	2019-03-16 12:33:07 +10:00
James Mills	e9c858d43f	Add CRC Checksum checks on reading values back	2019-03-16 12:16:23 +10:00
James Mills	120e854444	Improved error messages	2019-03-16 11:47:22 +10:00
James Mills	d2f44d1513	Fix a race condition + Use my fork of trie	2019-03-16 11:22:55 +10:00
James Mills	c0f178c4f7	Improved read/write performance by another ~2x by not calling Stat() on every read/write	2019-03-16 08:15:07 +10:00
James Mills	2585222830	Improve write performance by ~33% to 80,000 writes/sec buf reducing syscalls and using a bufio.Writer	2019-03-16 07:41:37 +10:00
James Mills	3f1d6635c4	Add prefix scan for keys using a Trie	2019-03-15 23:48:50 +10:00
James Mills	67840ffb57	Call Close() at end of sub-commands	2019-03-14 21:50:41 +10:00
James Mills	9f0a357ca0	Remove lock file on Close()	2019-03-14 21:50:23 +10:00
James Mills	52b6c74a21	Fixed compile error in CLI	2019-03-14 21:33:40 +10:00
James Mills	d24a01797a	Added WithMaxKeySize() and WithMaxValueSize() options	2019-03-14 21:31:23 +10:00
James Mills	bc8f6c6718	Change locking error message	2019-03-14 21:31:01 +10:00
James Mills	b6c212d60c	Refactored option handling	2019-03-14 21:24:31 +10:00
James Mills	3f1b90eb23	Update README.md	2019-03-14 18:18:57 +10:00