Zooko O'Whielacronx [Thu, 12 Feb 2009 03:11:29 +0000 (20:11 -0700)]
immutable repairer: errback any pending readers of DownUpConnectorwhen it runs out of bytes, and test that fact
Brian Warner [Thu, 12 Feb 2009 03:05:15 +0000 (20:05 -0700)]
figleaf_htmlizer: more rearranging, behavior should still be unchanged
Brian Warner [Thu, 12 Feb 2009 02:56:07 +0000 (19:56 -0700)]
figleaf_htmlizer: break it up into subfunctions, behavior should still be unchanged
Brian Warner [Thu, 12 Feb 2009 02:40:50 +0000 (19:40 -0700)]
figleaf_htmlizer: rewrite in class form, behavior should be the same as before
Brian Warner [Thu, 12 Feb 2009 02:16:43 +0000 (19:16 -0700)]
figleaf_htmlizer: rewrite with twisted.python.usage, remove logging: should behave the same as before
Brian Warner [Thu, 12 Feb 2009 02:05:42 +0000 (19:05 -0700)]
figleaf_htmlizer: expand tabs, fix to 4-space indents. No functional changes.
Brian Warner [Thu, 12 Feb 2009 01:09:13 +0000 (18:09 -0700)]
Makefile: include the figleaf pickle in the uploaded coverage data, for later differential analysis
Brian Warner [Thu, 12 Feb 2009 01:04:34 +0000 (18:04 -0700)]
Makefile: fix test-clean, by ignoring _appname.py and removing src/allmydata_tahoe.egg-info
Zooko O'Whielacronx [Thu, 12 Feb 2009 00:31:08 +0000 (17:31 -0700)]
setup: from the perspective of darcs, src/allmydata/_appname.py is a very boring file, indeed
Zooko O'Whielacronx [Thu, 12 Feb 2009 00:18:16 +0000 (17:18 -0700)]
versioning: include an "appname" in the application version string in the versioning protocol, and make that appname be controlled by setup.py
It is currently hardcoded in setup.py to be 'allmydata-tahoe'. Ticket #556 is to make it configurable by a runtime command-line argument to setup.py: "--appname=foo", but I suddenly wondered if we really wanted that and at the same time realized that we don't need that for tahoe-1.3.0 release, so this patch just hardcodes it in setup.py.
setup.py inspects a file named 'src/allmydata/_appname.py' and assert that it contains the string "__appname__ = 'allmydata-tahoe'", and creates it if it isn't already present. src/allmydata/__init__.py import _appname and reads __appname__ from it. The rest of the Python code imports allmydata and inspects "allmydata.__appname__", although actually every use it uses "allmydata.__full_version__" instead, where "allmydata.__full_version__" is created in src/allmydata/__init__.py to be:
__full_version__ = __appname + '-' + str(__version__).
All the code that emits an "application version string" when describing what version of a protocol it supports (introducer server, storage server, upload helper), or when describing itself in general (introducer client), usese allmydata.__full_version__.
This fixes ticket #556 at least well enough for tahoe-1.3.0 release.
Zooko O'Whielacronx [Wed, 11 Feb 2009 22:52:53 +0000 (15:52 -0700)]
setup: add new darcsver-1.2.1 which includes Brian's and Nils Durner's patch to ignore non-ascii chars in XML output
Brian Warner [Wed, 11 Feb 2009 22:28:30 +0000 (15:28 -0700)]
backupdb: cosmetic: capitalize the no-pysqlite instructions properly. Thanks to Terrell Russell for the catch.
Brian Warner [Wed, 11 Feb 2009 22:01:59 +0000 (15:01 -0700)]
test_repairer: disable repair-from-corruption tests until other things are improved well enough to make it useful
Brian Warner [Wed, 11 Feb 2009 21:43:52 +0000 (14:43 -0700)]
NEWS: explain limitations of the new repairer
Zooko O'Whielacronx [Wed, 11 Feb 2009 21:19:04 +0000 (14:19 -0700)]
setup: removed bundled darcsver-1.2.0
(I'm about to add a new bundled darcsver-1.2.1, but I want to see what the buildbots will do when there is no bundled darcsver present.)
Zooko O'Whielacronx [Wed, 11 Feb 2009 18:19:35 +0000 (11:19 -0700)]
trivial: whitespace
Brian Warner [Wed, 11 Feb 2009 21:14:53 +0000 (14:14 -0700)]
docs/known_issues: mention #615 javascript-vs-frames, for zooko to improve/rewrite
Brian Warner [Wed, 11 Feb 2009 06:39:38 +0000 (23:39 -0700)]
storage: make add-lease work, change default ownernum=1 since 0 is reserved to mean 'no lease here'
Brian Warner [Wed, 11 Feb 2009 06:37:56 +0000 (23:37 -0700)]
#620: storage: allow mutable shares to be deleted, with a writev where new_length=0
Brian Warner [Wed, 11 Feb 2009 03:37:09 +0000 (20:37 -0700)]
test_cli.Backup: insert some stalls to make sure two successive backups get distinct timestamps, avoiding intermittent failures
Brian Warner [Wed, 11 Feb 2009 01:49:10 +0000 (18:49 -0700)]
#619: make 'tahoe backup' complain and refuse to run if sqlite is unavailable and --no-backupdb is not passed
Brian Warner [Tue, 10 Feb 2009 22:00:25 +0000 (15:00 -0700)]
NEWS: point out that 'tahoe backup' requires a 1.3.0-or-later client node
Brian Warner [Tue, 10 Feb 2009 21:44:12 +0000 (14:44 -0700)]
tests: increase the default timeout for SystemTestMixin -based tests to 300 seconds, since our slower buildslaves sometimes take longer than the default 120s
Zooko O'Whielacronx [Tue, 10 Feb 2009 19:12:45 +0000 (12:12 -0700)]
immutable: repairer: add a simple test to exercise the "leftover" code path, fix the bug (and rename the variable "leftover" to "extra")
Zooko O'Whielacronx [Tue, 10 Feb 2009 18:02:27 +0000 (11:02 -0700)]
docs: not-quite-final version of relnotes.txt for tahoe-1.3.0
Zooko O'Whielacronx [Tue, 10 Feb 2009 18:02:19 +0000 (11:02 -0700)]
docs: small edit to about.html
Zooko O'Whielacronx [Tue, 10 Feb 2009 09:48:43 +0000 (02:48 -0700)]
docs: NEWS: move the most exciting items to the top, break them out of less exciting categories, update a couple of stale bits, and a touch of editing
Zooko O'Whielacronx [Tue, 10 Feb 2009 09:01:02 +0000 (02:01 -0700)]
docs: edit about.html
Zooko O'Whielacronx [Tue, 10 Feb 2009 07:56:47 +0000 (00:56 -0700)]
immutable: tighten preconditions -- you can write empty strings or read zero bytes, and add the first simple unit test of DownUpConnector
Zooko O'Whielacronx [Tue, 10 Feb 2009 06:53:48 +0000 (23:53 -0700)]
immutable: tests: the real WRITE_LEEWAY is 35 (it was a mistake to move it from 10 to 35 earlier -- I had seen a failure in which it took 35 times as many writes as I thought were optimal, but I misread and thought it took only 20 times as many)
Zooko O'Whielacronx [Tue, 10 Feb 2009 06:46:05 +0000 (23:46 -0700)]
immutable: defensive programming: assert that the encrypted readable gave you no more than the number of bytes you asked for
(There is a bug in the current DownUpConnector which can cause it to give more bytes than you asked for on one request, and then less on the next, effectively shifting some of the bytes to an earlier request, but I think this bug never gets triggered in practice.)
Zooko O'Whielacronx [Tue, 10 Feb 2009 06:44:21 +0000 (23:44 -0700)]
docs: suggest Python 2.5 -- Python 2.6 is not as well tested yet
Zooko O'Whielacronx [Tue, 10 Feb 2009 03:09:31 +0000 (20:09 -0700)]
immutable: tests: sigh, raise, again the limit of how many extra writes you can do and still pass this test
Obviously requiring the code under test to perform within some limit isn't very meaningful if we raise the limit whenever the test goes outside of it.
But I still don't want to remove the test code which measures how many writes (and, elsewhere, how many reads) a client does in order to fulfill these duties.
Let this number -- now 20 -- stand as an approximation of the inefficiency of our code divided by my mental model of how many operations are actually optimal for these duties.
Zooko O'Whielacronx [Tue, 10 Feb 2009 03:08:41 +0000 (20:08 -0700)]
immutable: tests: assert that verifier gives a clean bill of health after corruption and repair (the previous patch mistakenly did this only after deletion and repair), and also test whether deleting seven other shares and then downloading works. Also count the number of shares stored in the local filesystem.
Zooko O'Whielacronx [Tue, 10 Feb 2009 02:01:49 +0000 (19:01 -0700)]
immutable: test: add a test after attempting to repair from corruption: does a full verify run give the file a clean bill of health? If not, the you haven't successfully repaired it.
This will make the repairer tests more consistent -- less accidentally passing due to getting lucky.
Zooko O'Whielacronx [Tue, 10 Feb 2009 01:29:56 +0000 (18:29 -0700)]
immutable: tests: put shares back to their pristine condition in between each test of corrupting-and-repairing them
This is important, because if the repairer doesn't completely repair all kinds of corruption (as the current one doesn't), then the successive tests get messed up by assuming that the shares were uncorrupted when the test first set about to corrupt them.
Brian Warner [Mon, 9 Feb 2009 21:50:04 +0000 (14:50 -0700)]
upload: add a think-of-the-compatibility note to UploadResults
Brian Warner [Mon, 9 Feb 2009 21:45:43 +0000 (14:45 -0700)]
helper #609: uploading client should ignore old helper's UploadResults, which were in a different format
Brian Warner [Mon, 9 Feb 2009 09:34:00 +0000 (02:34 -0700)]
test_runner: skip all spawnProcess-using tests on cygwin, since spawnProcess just hangs forever
Brian Warner [Mon, 9 Feb 2009 09:26:17 +0000 (02:26 -0700)]
test_runner.py: revert the CreateNode section to using runner() inline, rather than spawning a process, to get more precise coverage
Brian Warner [Mon, 9 Feb 2009 02:56:02 +0000 (19:56 -0700)]
storage #596: announce 'tolerates-immutable-read-overrun' to the version announcement, to indicate that a read() on an immutable share where offset+length is beyond the end of the file will return a truncated string instead of raising an exception
Brian Warner [Mon, 9 Feb 2009 02:41:27 +0000 (19:41 -0700)]
test_upload: add test of maximum-immutable-share-size, to complete the last item of #538
Brian Warner [Mon, 9 Feb 2009 00:47:48 +0000 (17:47 -0700)]
docs/specifications: add an outline of the spec documents we'd like to have some day
Brian Warner [Sat, 7 Feb 2009 22:14:40 +0000 (15:14 -0700)]
test_cli.Backup: capture stderr when sqlite is unavailable
Zooko O'Whielacronx [Sat, 7 Feb 2009 23:33:21 +0000 (16:33 -0700)]
docs: add mac/README.txt to explain the first few basic facts about what the files are in this directory
Brian Warner [Sat, 7 Feb 2009 21:06:26 +0000 (14:06 -0700)]
test_repairer: wrap comments to 80cols, my laptop does not have a wide screen. No functional changes.
Brian Warner [Sat, 7 Feb 2009 21:04:39 +0000 (14:04 -0700)]
immutable/checker: wrap comments to 80cols, my laptop does not have a wide screen. No functional changes.
Brian Warner [Sat, 7 Feb 2009 20:38:17 +0000 (13:38 -0700)]
test/common.py: in share-layout-reading code, use '>L' consistently, since '>l' doesn't specify the signedness and the windows tests appear to be failing with an endianness-like problem (version==0x01000000). Also use binary mode when editing sharefiles
Brian Warner [Fri, 6 Feb 2009 09:17:53 +0000 (02:17 -0700)]
test_cli: increase timeout on test_backup, since our dapper buildslave is really slow
Brian Warner [Fri, 6 Feb 2009 08:34:01 +0000 (01:34 -0700)]
backupdb.py: catch OperationalError on duplicate-insert too, since pysqlite2 on dapper raises it instead of IntegrityError
Brian Warner [Fri, 6 Feb 2009 08:33:05 +0000 (01:33 -0700)]
test_backupdb.py: reset the check-timers after one step, otherwise a slow host can false-fail
Brian Warner [Fri, 6 Feb 2009 06:44:16 +0000 (23:44 -0700)]
NEWS: announce the #598 'tahoe backup' command
Brian Warner [Fri, 6 Feb 2009 05:14:45 +0000 (22:14 -0700)]
docs/CLI: document 'tahoe backup'
Brian Warner [Fri, 6 Feb 2009 05:10:42 +0000 (22:10 -0700)]
test_cli.backup: oops, fix test to work even when sqlite is unavailable
Brian Warner [Fri, 6 Feb 2009 05:07:01 +0000 (22:07 -0700)]
#598: add cli+backupdb tests, improve user display, update docs, move docs out of proposed/
Brian Warner [Fri, 6 Feb 2009 02:56:40 +0000 (19:56 -0700)]
#598: add backupdb to 'tahoe backup' command, enable it by default
Brian Warner [Fri, 6 Feb 2009 01:17:56 +0000 (18:17 -0700)]
add sqlite-based backupdb, for #598 and others (including 'tahoe cp'). Not enabled yet.
Zooko O'Whielacronx [Thu, 5 Feb 2009 16:29:23 +0000 (09:29 -0700)]
setup: require new bundled setuptools-0.6c12dev
Zooko O'Whielacronx [Thu, 5 Feb 2009 16:28:18 +0000 (09:28 -0700)]
setup: bundle setuptools-0.6c12dev (our own toothpick of setuptools) this version completes my patch to fix http://bugs.python.org/setuptools/issue54 , which is necessary for tahoe to build with --prefix=support without doing a lot of PYTHONPATH gymnastics around the call to setup.py
Zooko O'Whielacronx [Thu, 5 Feb 2009 16:27:58 +0000 (09:27 -0700)]
setup: remove old bundled setuptools-0.6c11dev (our own toothpick of setuptools)
Zooko O'Whielacronx [Thu, 5 Feb 2009 01:06:20 +0000 (18:06 -0700)]
setup: add a case to execute "python .../twistd.py" if "twistd" is not found
Zooko O'Whielacronx [Wed, 4 Feb 2009 22:38:40 +0000 (15:38 -0700)]
doc: specify Python >= 2.4.2
Zooko O'Whielacronx [Wed, 4 Feb 2009 20:32:22 +0000 (13:32 -0700)]
setup: merge recent patches that change the set of bundled tools in misc/dependencies/
Zooko O'Whielacronx [Wed, 4 Feb 2009 20:24:54 +0000 (13:24 -0700)]
setup: bundle new setuptools_trial
Zooko O'Whielacronx [Wed, 4 Feb 2009 20:24:42 +0000 (13:24 -0700)]
setup: remove old bundled setuptools_trial
Zooko O'Whielacronx [Wed, 4 Feb 2009 20:24:29 +0000 (13:24 -0700)]
setup: bundle new setuptools_darcs
Zooko O'Whielacronx [Wed, 4 Feb 2009 20:24:18 +0000 (13:24 -0700)]
setup: remove old bundled setuptools_darcs
Zooko O'Whielacronx [Wed, 4 Feb 2009 08:38:36 +0000 (01:38 -0700)]
roll back the upgrade of the bundled setuptools_trial from 0.5 to 0.5.1
Upgrading setuptools trial in that way leads to a very tricky problem in setuptools's handling of recursive installation of the build-time dependencies of build-time dependencies...
rolling back:
Tue Feb 3 22:17:18 MST 2009 zooko@zooko.com
* setup: bundle new setuptools_trial-0.5.1
A ./misc/dependencies/setuptools_trial-0.5.1.tar
Tue Feb 3 22:17:32 MST 2009 zooko@zooko.com
* setup: remove old bundled setuptools_trial
R ./misc/dependencies/setuptools_trial-0.5.tar
Zooko O'Whielacronx [Wed, 4 Feb 2009 06:24:05 +0000 (23:24 -0700)]
setup: remove a couple of horrible work-arounds in setup.py now that we rely on our own toothpick of setuptools which fixes those issues
also specify that we need our latest revision (11dev) of our toothpick of setuptools
also *always* setup_require setuptools_darcs at module import time. Formerly we added setup_require setuptools_darcs only if the PKG-INFO file were not already created. There is some weird, irreproducible bug to do with setuptool_darcs, and I guess that the fact that whether it is required or not depends on that autogenerated file might have something to do with it. Anyway, this is simpler.
Zooko O'Whielacronx [Wed, 4 Feb 2009 06:17:32 +0000 (23:17 -0700)]
setup: remove old bundled setuptools_trial
Zooko O'Whielacronx [Wed, 4 Feb 2009 06:17:18 +0000 (23:17 -0700)]
setup: bundle new setuptools_trial-0.5.1
Zooko O'Whielacronx [Wed, 4 Feb 2009 06:17:04 +0000 (23:17 -0700)]
setup: bundle new setuptools_darcs-1.2.5
Zooko O'Whielacronx [Wed, 4 Feb 2009 06:16:40 +0000 (23:16 -0700)]
setup: remove old bundled setuptools_darcs
Zooko O'Whielacronx [Wed, 4 Feb 2009 06:16:28 +0000 (23:16 -0700)]
setup: remove bundled darcsver in gzipped format
Zooko O'Whielacronx [Wed, 4 Feb 2009 06:16:17 +0000 (23:16 -0700)]
setup: bundle darcsver in ungzipped format
Zooko O'Whielacronx [Wed, 4 Feb 2009 04:34:10 +0000 (21:34 -0700)]
docs: mention issues using flogtool on Windows
Brian Warner [Wed, 4 Feb 2009 02:22:48 +0000 (19:22 -0700)]
webapi: add verifycap (spelled 'verify_url') to the t=json output on files and directories. Closes #559.
Zooko O'Whielacronx [Tue, 3 Feb 2009 05:23:52 +0000 (22:23 -0700)]
setup: stop trying to add specific dirs to site-dirs to work-around setuptools #17
Zooko O'Whielacronx [Tue, 3 Feb 2009 05:23:23 +0000 (22:23 -0700)]
setup: require and automatically use setuptools-0.6c11dev (our own toothpick of setuptools) which is bundled
Zooko O'Whielacronx [Tue, 3 Feb 2009 05:23:04 +0000 (22:23 -0700)]
setup: remove old bundled setuptools-0.6c10dev (our own toothpick of setuptools)
Zooko O'Whielacronx [Tue, 3 Feb 2009 05:19:50 +0000 (22:19 -0700)]
setup: bundle setuptools-0.6c11dev (our own toothpick of setuptools)
Hopefully this one fixes the issue with easy_install not searching the sys.path for packages that were requested to be installed, (setuptools #17), thus allowing us to do away with the "--site-dirs=" kludges, which are currently breaking some of our buildbots.
Brian Warner [Tue, 3 Feb 2009 04:12:32 +0000 (21:12 -0700)]
CLI: tahoe_check: stop escaping the JSON output when using --raw
Brian Warner [Tue, 3 Feb 2009 04:09:02 +0000 (21:09 -0700)]
#598: first cut of 'tahoe backup' command: no backupdb, but yes shared-unchanged-directories and Archives/TIMESTAMP and Latest/
Brian Warner [Tue, 3 Feb 2009 04:08:56 +0000 (21:08 -0700)]
CLI: move node-url -should-end-in-slash code into a common place, so other tools can rely upon it later
Brian Warner [Tue, 3 Feb 2009 04:07:20 +0000 (21:07 -0700)]
tahoe_ls: CLI command should return rc=0, not None
Zooko O'Whielacronx [Tue, 3 Feb 2009 01:09:55 +0000 (18:09 -0700)]
setup: add site-dirs for Debian and for some (?) Mac OS X
Brian Warner [Tue, 3 Feb 2009 00:36:57 +0000 (17:36 -0700)]
misc/dependencies: update setuptools_trial from 0.4 to 0.5
Brian Warner [Tue, 3 Feb 2009 00:03:33 +0000 (17:03 -0700)]
.darcs-boringfile: ignore bin/tahoe and bin/tahoe-script.py, since both are now generated
Brian Warner [Tue, 3 Feb 2009 00:03:07 +0000 (17:03 -0700)]
Makefile: fix 'clean' target to remove bin/tahoe and bin/tahoe-script.py, since both are now generated
Zooko O'Whielacronx [Sat, 31 Jan 2009 05:16:49 +0000 (22:16 -0700)]
storage: disable test_large_share again: my linux laptop has less than 4 GiB free
Brian Warner [Sat, 31 Jan 2009 02:32:05 +0000 (19:32 -0700)]
web/directory: add a link from readwrite directories to a read-only version, and fix the 'SI=xxx' header to actually use the storage index, not the writekey
Brian Warner [Sat, 31 Jan 2009 02:31:10 +0000 (19:31 -0700)]
uri: add abbrev_si() method, which returns the abbreviated storage index
Zooko O'Whielacronx [Fri, 30 Jan 2009 21:38:19 +0000 (14:38 -0700)]
setup: make sure you use darcsver whenever you are going to run trial
This fixes the bug Brian had where he ran "python ./setup.py trial" and the allmydata-tahoe version number came out as 0.0.0.
Zooko O'Whielacronx [Fri, 30 Jan 2009 05:31:33 +0000 (22:31 -0700)]
setup: require setuptools_trial >= 0.5, and delegate to it the job of deciding which Twisted reactor to use for the current platform
Brian Warner [Fri, 30 Jan 2009 11:25:36 +0000 (04:25 -0700)]
Makefile: use 'setup.py test' for test/quicktest targets (instead of
'setup.py trial'). 'setup.py trial' clobbers the tahoe .egg's PKG-INFO
"Version:" field (resetting it to 0.0.0), possibly because it isn't invoking
the darcsver subcommand that 'setup.py test' does before it runs the 'trial'
subcommand.
This slows down quicktest by another couple of seconds (see #591) and adds
more noise to its output, but without this change, 'make test' and 'make
quicktest' fail on test_runner (which spawns bin/tahoe as a subprocess, and
with a mangled Version:, the setuptools-based entry point script refuses to
recognize our source tree as a usable version of Tahoe).
Brian Warner [Fri, 30 Jan 2009 02:01:31 +0000 (19:01 -0700)]
Makefile: remove the obsolete stats-gatherer-run target
Zooko O'Whielacronx [Thu, 29 Jan 2009 20:59:52 +0000 (13:59 -0700)]
setup: remove the "build three times in a row" kludge now that #229 is fixed, and spell build "build" instead of "build_tahoe"
Zooko O'Whielacronx [Thu, 29 Jan 2009 19:56:40 +0000 (12:56 -0700)]
setup: require darcsver >= 1.2.0 and rely exclusively on darcsver to set the version string
Zooko O'Whielacronx [Thu, 29 Jan 2009 19:51:15 +0000 (12:51 -0700)]
setup: add new bundled darcsver-1.2.0
Zooko O'Whielacronx [Thu, 29 Jan 2009 19:51:02 +0000 (12:51 -0700)]
setup: remove old bundled darcsver-1.1.8