Make backupdb use dbutil.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Makefile: have 'make tmpfstest' unmount and remove stale temp directories.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Makefile: the timing for 'make tmpfstest' should exclude filesystem
mounting/unmounting and entering the password if needed.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Makefile: allow tmpfs size to be more easily overridden, and use 500 MiB by default (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

test_runner.py: add test for 'tahoe debug trial'.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Fixes to tests. Some tests are applied to multiple backends.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Fixes to test infrastructure.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Miscellaneous corrections and additions.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to crawler classes (ShareCrawler and AccountingCrawler).
Pass in a Clock to allow (in theory) deterministic testing, although this isn't used yet by tests.
Simplify the generic ShareCrawler code by not attempting to track state during processing
of a single prefix.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to fileutil.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Fix bugs in Accountant.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Simplify Account.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to debug.py.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Changes to node classes (Node, Client and StorageServer).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Changes to Bucket{Reader,Writer} and disk backend (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Changes and additions to interface documentation.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add new files for cloud merge (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Move BucketWriter and BucketReader to storage/bucket.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Move code around and add new directories for cloud backend merge.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Add dependency on our fork of txAWS (0.2.1.post5).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

.gitignore: changes to facilitate cloud backend merge (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add documentation for each storage backend (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add test_leasedb.py.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

leasedb: use a semantic primary key (storage_index, shnum, account_id), rather than an integer, for the leases table.
Take advantage of this to simplify add_or_renew_leases.
Fix a bug in add_starter_lease (which is not used yet) when a starter lease already exists.
Clean up leftover accesses to self._dirty.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Rename 'buckets' to 'sharesets' on storage status page.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove unused files storage/lease.py and storage/expirer.py.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Main leasedb changes (rebased).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Remove the 'original-*' and 'configured-*' lease crawler output that won't be supported by the accounting crawler.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Cosmetics.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

test_storage.py: ss -> server for cases that will remain a server after the server/account split.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Asyncify crawlers. Note that this breaks tests for the LeaseCrawler
(which is going away, to be replaced by the AccountingCrawler).

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

scripts/debug.py: remove display of lease information and secrets.
This version replaces the expiration field with '-' instead of '0', per Zooko's comments.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Remove the [storage]expire.{mutable,immutable} options.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Remove support for [storage]debug_discard option.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add new files for leasedb.

Authors: Brian Warner <warner@lothar.com> and David-Sarah Hopwood
Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Changes to specification of add_lease and renew_lease in RIStorageServer.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

docs/garbage-collection.rst: update text for leasedb changes.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

util/fileutil.py: add get_used_space. This version does not use FilePath.

Signed-off-by: David-Sarah Hopwood <david-sarah@jacaranda.org>

Change some instances of "filesystem" that were missed to "file store".

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

update docs for 1.10.2 release

add a pypy that is allowed to fail

update NEWS with recent changes

quickstart: Mention python-dev package for Python.h

Closes ticket:2328.

Merge branch '2447-cp-overwrite'

test_cli_cp: minor cleanup

cp: error on target-filename collisions, rather than overwrite

Closes ticket:2447

unit test for my desired behavior

Merge branch '2436.less-scary.3'

This reverts and improves the earlier fix for ticket:2436

test_version.py: minor cleanup.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Don't show scary diagnostic warnings from --version[-and-path] (corrected). refs ticket:2436

The previous version would incorrectly add to the output of
get_package_versions_string each time it was called.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Revert "Don't show scary diagnostic warnings from --version[-and-path]"

This reverts commit 431728f8f854e02d9ab2f731675f12ce18cda122.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

mutable/retrieve.py: remove all bare assert()s

Replace them with _assert() or precondition(). Closes ticket:2462.

improve HTTP byte-range docs

webapi.rst: document Range: header on GET requests

closes ticket#2460

IReadable.read(): document out-of-range errors

closes ticket#2461

Merge branch 'sdmf-partial-2'

This cleans up the mutable-retrieve code a bit, and should fix some
corner cases where an offset/size combinations that reads the last byte
of the file (but not the first) could cause an assert to fire, making
the download hang. Should address ticket:2459 and ticket:2462.

test_mutable: more enhancements

mutable/retrieve.py: rewrite partial-read handling

This should tolerate offset/size combinations that read the last byte of
the file, something which was broken before. It quits early in the case
of zero-byte reads, to simplify the resulting "which segments do I need"
logic. Probably addresses ticket:2459.

enhance SDMF/MDMF tests: zero-length files, size=None

Add tests for SDMF partial reads. refs #2459

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Remove some bare asserts in retrieve.py (there are still quite a few left). refs #2462

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge pull request #176 from tahoe-lafs/2436.less-scary.0

Don't show scary diagnostic warnings from --version[-and-path] if the only problem is that extra packages are found by pkg_resources.

Minor code cleanup in __init__.py.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Don't show scary diagnostic warnings from --version[-and-path]
if the only problem is that extra packages are found by pkg_resources.
fixes ticket:2436

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Change the separator character in the Magic Folder user interface design doc from '|' to '+'.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Eliminate mock dependency.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add Namespace utility class. (This is already used in the leasedb/cloud backend branch.)

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Makefile upload-osx-pkg: mention where the packages go

Bump pyasn1 requirement to >= 0.1.8.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Blacklist mock > 1.0.1.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

test/check_load: hush (newer) pyflakes

pyflakes-0.9.2 gets confused about list comprehensions, so rename the
loop variable even though we'd deleted that name already.

test_web: avoid IP autodetection, was causing dirty reactor

docs/how-to-release: add more wiki pages to update

add test-pip-install

release checklist: use signed git tag next time

README: update copyright date

update docs for 1.10.1 release

Dockerfile: git-pull more history, to get a version string

The Docker build process seems to use 'git pull --depth=1', which
doesn't fetch enough history to see a tag, which means tahoe's setup.py
can't compute a version (then 'tahoe --version' reports "unknown").
Pulling a hundred commits should be enough to see a tag without making
the resulting image unnecessarily large. If we go more than 100 commits
between tags, we'll need to increase this, or come up with something
smarter.

Finish user-interface.rst. fixes ticket:2443

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Make a start on the Magic Folder Objective 5 (user interface design) doc. refs ticket:2443

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

NEWS.rst: fix missing link.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

NEWS.rst for 1.10.1: links, formatting and nitpicks.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

NEWS: finish 1.10.1 edits. Good enough for beta1.

NEWS: rearrange

NEWS: remove many non-user-visible items

NEWS: first editing pass

NEWS: classify changes as user-visible or not

NEWS.rst: update to include latest changes

move Dockerfile back to root. DockerHub can't handle it elsewhere.

It looks like DockerHub's automated builds only have access to the
subtree of the source checkout at+below the Dockerfile. Putting the
Dockerfile in misc/build_helpers/ meant that the build process only had
access to misc/build_helpers/, not the full source tree.

Dockerfile: fix ADD path to deal with non-root location

move Dockerfile down ito misc/build_helpers/

Magic Folder design doc: rename "parent DMD" to "collective directory", and don't assume it is mutable.
(An immutable collective directory could be interesting for some use cases, and is no more difficult to support.)

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Remove packages from not_import_versionable that we now can always version.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add OpenSSL version check and tests. refs ticket:2215

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

remote-to-local-sync.rst: fix a minor error.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Magic Folder: add remote-to-local sync design doc.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge branch 'globalopts2'

Improves CLI rendering of --help for global options. Closes ticket:2233.

wrap long lines, and tolerate various-width wrappings of the --help output

test_cli.Help was too sensitive to the way that the --help output was
wrapped, which caused failures on travis when COLUMNS= was set low and
the expected strings were split across separate lines.

cli: improve formatting of all commands

Also:

* do some light refactoring of create-client/node
* make it clear that these commands' --basedir options do the same as
  the global --node-directory option
* use "global-options" instead of "global-opts"

scripts: improve rendering of synopsis/usage

Subcommands "--help" is now rendered as:

```
 tahoe [global-options] COMMAND [options] ARGS
 (use 'tahoe --help' to view global options)
 USAGE (flags/options)
 DESCRIPTION
 DESCRIPTION_UNWRAPPED
```

The new .description and .description_unwrapped fields allow
commands (subclasses of twisted.python.usage.Usage) better control over
how their explanations are rendered: the old .longdesc field was wrapped
unpleasantly.