tahoe-lafs/tahoe-lafs.git
15 years agoimmutable/checker.py: trap ShareVersionIncompatible too. Also, use f.check
Brian Warner [Tue, 24 Feb 2009 05:14:05 +0000 (22:14 -0700)]
immutable/checker.py: trap ShareVersionIncompatible too. Also, use f.check
instead of examining the value returned by f.trap, because the latter appears
to squash exception types down into their base classes (i.e. since
ShareVersionIncompatible is a subclass of LayoutInvalid,
f.trap(Failure(ShareVersionIncompatible)) == LayoutInvalid).

All this resulted in 'incompatible' shares being misclassified as 'corrupt'.

15 years agoimmutable/layout.py: wrap to 80 cols, no functional changes
Brian Warner [Tue, 24 Feb 2009 01:58:37 +0000 (18:58 -0700)]
immutable/layout.py: wrap to 80 cols, no functional changes

15 years agotest_repairer: change Repairer to use much-faster no_network.GridTestMixin. As a...
Brian Warner [Tue, 24 Feb 2009 00:42:27 +0000 (17:42 -0700)]
test_repairer: change Repairer to use much-faster no_network.GridTestMixin. As a side-effect, fix what I think was a bug: some of the assert-minimal-effort-expended checks were mixing write counts and allocate counts

15 years agotest/no_network.py: add a basic stats provider
Brian Warner [Tue, 24 Feb 2009 00:39:37 +0000 (17:39 -0700)]
test/no_network.py: add a basic stats provider

15 years agotests: stop using setUpClass/tearDownClass, since they've been deprecated in Twisted...
Brian Warner [Mon, 23 Feb 2009 21:43:12 +0000 (14:43 -0700)]
tests: stop using setUpClass/tearDownClass, since they've been deprecated in Twisted-8.2.0

15 years agotest_checker: improve test coverage for checker results
Brian Warner [Mon, 23 Feb 2009 21:19:43 +0000 (14:19 -0700)]
test_checker: improve test coverage for checker results

15 years agoFixed tests again so they will pass on windows.
Alberto Berti [Mon, 23 Feb 2009 01:35:02 +0000 (18:35 -0700)]
Fixed tests again so they will pass on windows.

15 years agomisc/*: remove RuntimeError too
Brian Warner [Mon, 23 Feb 2009 00:34:01 +0000 (17:34 -0700)]
misc/*: remove RuntimeError too

15 years agoscripts: stop using RuntimeError, for #639
Brian Warner [Mon, 23 Feb 2009 00:31:06 +0000 (17:31 -0700)]
scripts: stop using RuntimeError, for #639

15 years agomutable/publish: stop using RuntimeError, for #639
Brian Warner [Mon, 23 Feb 2009 00:30:56 +0000 (17:30 -0700)]
mutable/publish: stop using RuntimeError, for #639

15 years agoremove more RuntimeError from unit tests, for #639
Brian Warner [Mon, 23 Feb 2009 00:28:55 +0000 (17:28 -0700)]
remove more RuntimeError from unit tests, for #639

15 years agostop using RuntimeError in unit tests, for #639
Brian Warner [Mon, 23 Feb 2009 00:27:22 +0000 (17:27 -0700)]
stop using RuntimeError in unit tests, for #639

15 years agoftpd/sftpd: stop using RuntimeError, for #639
Brian Warner [Mon, 23 Feb 2009 00:24:26 +0000 (17:24 -0700)]
ftpd/sftpd: stop using RuntimeError, for #639

15 years agoAdded tests for the cse when listdir is an iterator
Alberto Berti [Sun, 22 Feb 2009 23:43:56 +0000 (16:43 -0700)]
Added tests for the cse when listdir is an iterator

15 years agoFixed tests so that they pass also on buildbots.
Alberto Berti [Sun, 22 Feb 2009 23:43:11 +0000 (16:43 -0700)]
Fixed tests so that they pass also on buildbots.

15 years agoUse failUnlessEqual instead of failUnless(a == b)
Alberto Berti [Sun, 22 Feb 2009 23:42:14 +0000 (16:42 -0700)]
Use failUnlessEqual instead of failUnless(a == b)

15 years agoBetter implementation of filtering algorithm.
Alberto Berti [Sun, 22 Feb 2009 23:40:49 +0000 (16:40 -0700)]
Better implementation of filtering algorithm.

15 years agoRemoved '.hgrags' from vcs excludes
Alberto Berti [Sun, 22 Feb 2009 23:39:46 +0000 (16:39 -0700)]
Removed '.hgrags' from vcs excludes

15 years agodocs: CREDITS to Alberto Berti
Zooko O'Whielacronx [Sun, 22 Feb 2009 20:33:14 +0000 (13:33 -0700)]
docs: CREDITS to Alberto Berti

15 years agoAdded tests for the --exclude* options of backup command.
Alberto Berti [Sun, 22 Feb 2009 17:51:06 +0000 (10:51 -0700)]
Added tests for the --exclude* options of backup command.

15 years agoAdded --exclude, --exclude-from and --exclude-vcs options to backup command.
Alberto Berti [Sun, 22 Feb 2009 18:08:29 +0000 (11:08 -0700)]
Added --exclude, --exclude-from and --exclude-vcs options to backup command.

It is still impossible to specify absolute exclusion path, only
relative. I must check with tar or rsync how they allow them to be
specified.

15 years agoRaise a more explanatory exception for errors encountered during backup processing.
Alberto Berti [Sun, 22 Feb 2009 18:02:52 +0000 (11:02 -0700)]
Raise a more explanatory exception for errors encountered during backup processing.

15 years agoAdded tests for the fixed alias related command's synopsis
Alberto Berti [Sun, 22 Feb 2009 17:37:32 +0000 (10:37 -0700)]
Added tests for the fixed alias related command's synopsis

15 years agoAdd missing synopsis and descriptions for alias commands.
Alberto Berti [Sat, 21 Feb 2009 01:31:06 +0000 (18:31 -0700)]
Add missing synopsis and descriptions for alias commands.

15 years agodocs: move many specification-like documents into specifications/
Brian Warner [Sun, 22 Feb 2009 06:40:54 +0000 (23:40 -0700)]
docs: move many specification-like documents into specifications/

15 years agotest_web.py: increase test coverage of web.status.plural()
Brian Warner [Sun, 22 Feb 2009 01:01:16 +0000 (18:01 -0700)]
test_web.py: increase test coverage of web.status.plural()

15 years agocrawler: fix performance problems: only save state once per timeslice (not after...
Brian Warner [Sat, 21 Feb 2009 21:56:49 +0000 (14:56 -0700)]
crawler: fix performance problems: only save state once per timeslice (not after every bucket), don't start the crawler until 5 minutes after node startup

15 years agotest_system: oops, don't assume that all files in storage/ are in a deep storage...
Brian Warner [Sat, 21 Feb 2009 07:17:10 +0000 (00:17 -0700)]
test_system: oops, don't assume that all files in storage/ are in a deep storage/shares/prefix/si/shnum path, since now the crawler pickle has a short path

15 years agocrawler: tolerate low-resolution system clocks (i.e. windows)
Brian Warner [Sat, 21 Feb 2009 07:15:33 +0000 (00:15 -0700)]
crawler: tolerate low-resolution system clocks (i.e. windows)

15 years agoBucketCountingCrawler: store just the count, not cycle+count, since it's too easy...
Brian Warner [Sat, 21 Feb 2009 04:58:31 +0000 (21:58 -0700)]
BucketCountingCrawler: store just the count, not cycle+count, since it's too easy to make usage mistakes otherwise

15 years agotest_storage: startService the server, as is now the standard practice
Brian Warner [Sat, 21 Feb 2009 04:57:55 +0000 (21:57 -0700)]
test_storage: startService the server, as is now the standard practice

15 years agocrawler: load state from the pickle in init, rather than waiting until startService...
Brian Warner [Sat, 21 Feb 2009 04:57:20 +0000 (21:57 -0700)]
crawler: load state from the pickle in init, rather than waiting until startService, so get_state() can be called early

15 years agoBucketCountingCrawler: rename status and state keys to use 'bucket' instead of 'share...
Brian Warner [Sat, 21 Feb 2009 04:46:06 +0000 (21:46 -0700)]
BucketCountingCrawler: rename status and state keys to use 'bucket' instead of 'share', because the former is more accurate

15 years agostorage: also report space-free-for-root and space-free-for-nonroot, since that helps...
Brian Warner [Sat, 21 Feb 2009 04:28:56 +0000 (21:28 -0700)]
storage: also report space-free-for-root and space-free-for-nonroot, since that helps users understand the space-left-for-tahoe number better

15 years agostorage: add bucket-counting share crawler, add its output (number of files+directori...
Brian Warner [Sat, 21 Feb 2009 04:04:08 +0000 (21:04 -0700)]
storage: add bucket-counting share crawler, add its output (number of files+directories maintained by a storage server) and status to the webapi /storage page

15 years agostorage: move si_b2a/si_a2b/storage_index_to_dir out of server.py and into common.py
Brian Warner [Sat, 21 Feb 2009 04:03:09 +0000 (21:03 -0700)]
storage: move si_b2a/si_a2b/storage_index_to_dir out of server.py and into common.py

15 years agocrawler: add get_progress, clean up get_state
Brian Warner [Sat, 21 Feb 2009 01:27:43 +0000 (18:27 -0700)]
crawler: add get_progress, clean up get_state

15 years agoweb/storage: make sure we can handle platforms without os.statvfs too
Brian Warner [Fri, 20 Feb 2009 23:03:53 +0000 (16:03 -0700)]
web/storage: make sure we can handle platforms without os.statvfs too

15 years agocrawler: provide for one-shot crawlers, which stop after their first full cycle,...
Brian Warner [Fri, 20 Feb 2009 22:19:11 +0000 (15:19 -0700)]
crawler: provide for one-shot crawlers, which stop after their first full cycle, for share-upgraders and database-populaters

15 years agoweb: add Storage status page, improve tests
Brian Warner [Fri, 20 Feb 2009 21:29:26 +0000 (14:29 -0700)]
web: add Storage status page, improve tests

15 years agostorage: include reserved_space in stats
Brian Warner [Fri, 20 Feb 2009 21:29:20 +0000 (14:29 -0700)]
storage: include reserved_space in stats

15 years agoweb/check_results: sort share identifiers in the sharemap display
Brian Warner [Fri, 20 Feb 2009 19:29:22 +0000 (12:29 -0700)]
web/check_results: sort share identifiers in the sharemap display

15 years agowebapi: pass client through constructor arguments, remove IClient, should make it...
Brian Warner [Fri, 20 Feb 2009 19:15:54 +0000 (12:15 -0700)]
webapi: pass client through constructor arguments, remove IClient, should make it easier to test web renderers in isolation

15 years agotest/no_network: do startService on the storage servers, make it easier to customize...
Brian Warner [Fri, 20 Feb 2009 03:22:54 +0000 (20:22 -0700)]
test/no_network: do startService on the storage servers, make it easier to customize the storage servers

15 years agocrawler: modify API to support upcoming bucket-counting crawler
Brian Warner [Fri, 20 Feb 2009 02:31:42 +0000 (19:31 -0700)]
crawler: modify API to support upcoming bucket-counting crawler

15 years agotest_backupdb: make the not-a-database file larger, since the older sqlite-2.3.2...
Brian Warner [Fri, 20 Feb 2009 01:04:09 +0000 (18:04 -0700)]
test_backupdb: make the not-a-database file larger, since the older sqlite-2.3.2 on OS-X is easily fooled

15 years agoweb/reliability: add parameter descriptions, adapted from a patch from Terrell Russell.
Brian Warner [Thu, 19 Feb 2009 23:29:18 +0000 (16:29 -0700)]
web/reliability: add parameter descriptions, adapted from a patch from Terrell Russell.

15 years agotest_crawler: hush pyflakes
Brian Warner [Thu, 19 Feb 2009 21:23:40 +0000 (14:23 -0700)]
test_crawler: hush pyflakes

15 years agotest_crawler: disable the percentage-of-cpu-used test, since it is too unreliable...
Brian Warner [Thu, 19 Feb 2009 21:16:54 +0000 (14:16 -0700)]
test_crawler: disable the percentage-of-cpu-used test, since it is too unreliable on our slow buildslaves. But leave the code in place for developers to run by hand.

15 years agoreliability.py: fix the numpy conversion, it was completely broken. Thanks to Terrell...
Brian Warner [Thu, 19 Feb 2009 20:55:15 +0000 (13:55 -0700)]
reliability.py: fix the numpy conversion, it was completely broken. Thanks to Terrell Russell for the help.

15 years agoreliability: switch to NumPy, since Numeric is deprecated
Brian Warner [Thu, 19 Feb 2009 08:44:35 +0000 (01:44 -0700)]
reliability: switch to NumPy, since Numeric is deprecated

15 years agosetup.py: fix pyflakes complaints
Brian Warner [Thu, 19 Feb 2009 08:36:43 +0000 (01:36 -0700)]
setup.py: fix pyflakes complaints

15 years agomove show-tool-versions out of setup.py and into a separate script in misc/ , since...
Brian Warner [Thu, 19 Feb 2009 08:35:58 +0000 (01:35 -0700)]
move show-tool-versions out of setup.py and into a separate script in misc/ , since setuptools is trying to build and install a bunch of stuff first

15 years agotest_crawler: don't require >=1 cycle on cygwin
Brian Warner [Thu, 19 Feb 2009 07:58:18 +0000 (00:58 -0700)]
test_crawler: don't require >=1 cycle on cygwin

15 years agosetup.py: add show_tool_versions command, for the benefit of a new buildbot step
Brian Warner [Thu, 19 Feb 2009 07:24:36 +0000 (00:24 -0700)]
setup.py: add show_tool_versions command, for the benefit of a new buildbot step

15 years agosetup.py: wrap to 80 cols, no functional changes
Brian Warner [Thu, 19 Feb 2009 06:57:51 +0000 (23:57 -0700)]
setup.py: wrap to 80 cols, no functional changes

15 years agocrawler: use fileutil.move_info_place in preference to our own version
Brian Warner [Thu, 19 Feb 2009 06:13:42 +0000 (23:13 -0700)]
crawler: use fileutil.move_info_place in preference to our own version

15 years agofileutil: add move_into_place(), to perform the standard unix trick of atomically...
Brian Warner [Thu, 19 Feb 2009 06:13:10 +0000 (23:13 -0700)]
fileutil: add move_into_place(), to perform the standard unix trick of atomically replacing a file, with a fallback for windows

15 years agocrawler: fix problems on windows and our slow cygwin slave
Brian Warner [Thu, 19 Feb 2009 05:24:31 +0000 (22:24 -0700)]
crawler: fix problems on windows and our slow cygwin slave

15 years ago#633: first version of a rate-limited interruptable share-crawler
Brian Warner [Thu, 19 Feb 2009 04:46:33 +0000 (21:46 -0700)]
#633: first version of a rate-limited interruptable share-crawler

15 years agochange StorageServer to take nodeid in the constructor, instead of assigning it later...
Brian Warner [Wed, 18 Feb 2009 23:23:01 +0000 (16:23 -0700)]
change StorageServer to take nodeid in the constructor, instead of assigning it later, since it's cleaner and because the original problem (Tubs not being ready until later) went away

15 years agotest_system: split off checker tests to test_deepcheck.py, this file is too big
Brian Warner [Wed, 18 Feb 2009 22:42:34 +0000 (15:42 -0700)]
test_system: split off checker tests to test_deepcheck.py, this file is too big

15 years agobreak storage.py into smaller pieces in storage/*.py . No behavioral changes.
Brian Warner [Wed, 18 Feb 2009 21:46:55 +0000 (14:46 -0700)]
break storage.py into smaller pieces in storage/*.py . No behavioral changes.

15 years agoimmutable/layout: minor change to repr name
Brian Warner [Wed, 18 Feb 2009 21:46:48 +0000 (14:46 -0700)]
immutable/layout: minor change to repr name

15 years agodocs: add lease-tradeoffs diagram
Brian Warner [Wed, 18 Feb 2009 21:41:37 +0000 (14:41 -0700)]
docs: add lease-tradeoffs diagram

15 years agointerfaces.py: allow add/renew/cancel-lease to return Any, so that 1.3.1 clients...
Brian Warner [Wed, 18 Feb 2009 20:29:03 +0000 (13:29 -0700)]
interfaces.py: allow add/renew/cancel-lease to return Any, so that 1.3.1 clients (the first to use these calls) can tolerate future storage servers which might return something other than None

15 years agodocs/debian.txt: minor edit
Brian Warner [Wed, 18 Feb 2009 04:22:12 +0000 (21:22 -0700)]
docs/debian.txt: minor edit

15 years agoadd --add-lease to 'tahoe check', 'tahoe deep-check', and webapi.
Brian Warner [Wed, 18 Feb 2009 02:32:43 +0000 (19:32 -0700)]
add --add-lease to 'tahoe check', 'tahoe deep-check', and webapi.

15 years agochange RIStorageServer.remote_add_lease to exit silently in case of no-such-bucket...
Brian Warner [Wed, 18 Feb 2009 02:30:53 +0000 (19:30 -0700)]
change RIStorageServer.remote_add_lease to exit silently in case of no-such-bucket, instead of raising IndexError, because that makes the upcoming --add-lease feature faster and less noisy

15 years agoCLI #590: convert 'tahoe deep-check' to streaming form, improve display, add tests
Brian Warner [Wed, 18 Feb 2009 00:15:11 +0000 (17:15 -0700)]
CLI #590: convert 'tahoe deep-check' to streaming form, improve display, add tests

15 years agointerfaces.py: document behavior of add_lease/renew_lease/cancel_lease, before I...
Brian Warner [Tue, 17 Feb 2009 20:48:09 +0000 (13:48 -0700)]
interfaces.py: document behavior of add_lease/renew_lease/cancel_lease, before I change it

15 years agotest_backupdb: improve error messages if the test fails
Brian Warner [Tue, 17 Feb 2009 18:08:38 +0000 (11:08 -0700)]
test_backupdb: improve error messages if the test fails

15 years agowebapi #590: add streaming deep-check. Still need a CLI tool to use it.
Brian Warner [Tue, 17 Feb 2009 06:35:53 +0000 (23:35 -0700)]
webapi #590: add streaming deep-check. Still need a CLI tool to use it.

15 years agotest_web.Grid: change the CHECK() function to make it easier to test t= values with...
Brian Warner [Tue, 17 Feb 2009 06:00:34 +0000 (23:00 -0700)]
test_web.Grid: change the CHECK() function to make it easier to test t= values with hyphens in them

15 years agotest_web: improve checker-results coverage with a no-network -based test, enhance...
Brian Warner [Tue, 17 Feb 2009 05:12:42 +0000 (22:12 -0700)]
test_web: improve checker-results coverage with a no-network -based test, enhance no-network harness to assist, fix some bugs in web/check_results.py that were exposed

15 years agoweb: fix handling of reliability page when Numeric is not available
Brian Warner [Tue, 17 Feb 2009 02:56:58 +0000 (19:56 -0700)]
web: fix handling of reliability page when Numeric is not available

15 years agotest/no_network: update comments with setup timing: no_network takes 50ms, SystemTest...
Brian Warner [Tue, 17 Feb 2009 01:06:43 +0000 (18:06 -0700)]
test/no_network: update comments with setup timing: no_network takes 50ms, SystemTestMixin takes 2s (on my laptop)

15 years agotest_upload: rewrite in terms of no-network GridTestMixin, improve no_network.py...
Brian Warner [Tue, 17 Feb 2009 00:44:57 +0000 (17:44 -0700)]
test_upload: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary

15 years agotest_download: rewrite in terms of no-network GridTestMixin, improve no_network.py...
Brian Warner [Tue, 17 Feb 2009 00:36:58 +0000 (17:36 -0700)]
test_download: rewrite in terms of no-network GridTestMixin, improve no_network.py as necessary

15 years agotest_dirnode.py: convert Deleter to new no-network gridtest
Brian Warner [Tue, 17 Feb 2009 00:23:48 +0000 (17:23 -0700)]
test_dirnode.py: convert Deleter to new no-network gridtest

15 years agotest_cli.py: modify to use the new 'no-network' gridtest instead of SystemTestMixin...
Brian Warner [Tue, 17 Feb 2009 00:20:05 +0000 (17:20 -0700)]
test_cli.py: modify to use the new 'no-network' gridtest instead of SystemTestMixin, which speeds it up from 73s to 43s on my system

15 years agotests: fix no_network framework to work with upload/download and checker
Brian Warner [Tue, 17 Feb 2009 00:19:47 +0000 (17:19 -0700)]
tests: fix no_network framework to work with upload/download and checker

15 years agoclient.py: improve docstring
Brian Warner [Tue, 17 Feb 2009 00:15:32 +0000 (17:15 -0700)]
client.py: improve docstring

15 years agotest_cli: add test coverage for help strings
Brian Warner [Mon, 16 Feb 2009 22:08:33 +0000 (15:08 -0700)]
test_cli: add test coverage for help strings

15 years agotest/no_network: new test harness, like system-test but doesn't use the network so...
Brian Warner [Mon, 16 Feb 2009 21:58:44 +0000 (14:58 -0700)]
test/no_network: new test harness, like system-test but doesn't use the network so it's faster

15 years agointerfaces.py: minor docstring edit
Brian Warner [Mon, 16 Feb 2009 21:58:16 +0000 (14:58 -0700)]
interfaces.py: minor docstring edit

15 years agosetup: if you sdist_dsc (to produce the input files for dpkg-buildpackage) then run...
Zooko O'Whielacronx [Mon, 16 Feb 2009 21:15:58 +0000 (14:15 -0700)]
setup: if you sdist_dsc (to produce the input files for dpkg-buildpackage) then run darcsver first

15 years agodoc: a few edits to docs made after the 1.3.0 release
Zooko O'Whielacronx [Mon, 16 Feb 2009 21:15:39 +0000 (14:15 -0700)]
doc: a few edits to docs made after the 1.3.0 release

15 years agotest_cli: improve test coverage slightly
Brian Warner [Mon, 16 Feb 2009 04:04:51 +0000 (21:04 -0700)]
test_cli: improve test coverage slightly

15 years agotest_util: get almost full test coverage of dictutil, starting with the original...
Brian Warner [Mon, 16 Feb 2009 03:32:10 +0000 (20:32 -0700)]
test_util: get almost full test coverage of dictutil, starting with the original pyutil tests as a base. The remaining three uncovered lines involve funny cases of ValueOrderedDict that I can't figure out how to get at

15 years agoprovisioning/reliability: add tests, hush pyflakes, remove dead code, fix web links
Brian Warner [Sun, 15 Feb 2009 23:24:51 +0000 (16:24 -0700)]
provisioning/reliability: add tests, hush pyflakes, remove dead code, fix web links

15 years agoutil/statistics: add tests, fix mean_repair_cost
Brian Warner [Sun, 15 Feb 2009 23:23:26 +0000 (16:23 -0700)]
util/statistics: add tests, fix mean_repair_cost

15 years agotest_repairer: hush pyflakes
Brian Warner [Sun, 15 Feb 2009 23:23:10 +0000 (16:23 -0700)]
test_repairer: hush pyflakes

15 years agolossmodel.lyx: move draft paper into docs/proposed/, since it's unfinished
Brian Warner [Sun, 15 Feb 2009 23:19:05 +0000 (16:19 -0700)]
lossmodel.lyx: move draft paper into docs/proposed/, since it's unfinished

15 years agobuild a 'reliability' web page, with a simulation of file decay and repair over time
Brian Warner [Sat, 14 Feb 2009 00:42:34 +0000 (17:42 -0700)]
build a 'reliability' web page, with a simulation of file decay and repair over time

15 years agoMore lossmodel work, on repair.
Shawn Willden [Fri, 16 Jan 2009 03:56:48 +0000 (20:56 -0700)]
More lossmodel work, on repair.

15 years agoLoss model work (temp1)
Shawn Willden [Thu, 15 Jan 2009 04:00:58 +0000 (21:00 -0700)]
Loss model work (temp1)

15 years agoStatistics module
Shawn Willden [Wed, 14 Jan 2009 03:12:35 +0000 (20:12 -0700)]
Statistics module

Added a statistics module for calculating various facets of
share survival statistics.

15 years agodocs: relnotes-short.txt
Zooko O'Whielacronx [Sun, 15 Feb 2009 17:35:10 +0000 (10:35 -0700)]
docs: relnotes-short.txt
linkedin.com imposed a strict limit on the number of characters I could post.  This forced me to prune and prune and edit and edit until relnotes.txt was a quarter of its former size.  Here's the short version.

15 years agoTAG allmydata-tahoe-1.3.0 allmydata-tahoe-1.3.0
Zooko O'Whielacronx [Sat, 14 Feb 2009 01:05:56 +0000 (18:05 -0700)]
TAG allmydata-tahoe-1.3.0