Brian Warner [Wed, 14 May 2008 21:32:21 +0000 (14:32 -0700)]
webish: add /file links, change directory page to use them. This fixes filenames in wget. Closes #221.
Brian Warner [Wed, 14 May 2008 20:55:47 +0000 (13:55 -0700)]
introweb.py: tolerate foolscap>=0.2.6, which changed the internals of .locationHints
Brian Warner [Wed, 14 May 2008 20:10:41 +0000 (13:10 -0700)]
test_mutable.py: test replacing a file that has one new outlier share present: closes #272
Brian Warner [Sat, 10 May 2008 01:06:29 +0000 (18:06 -0700)]
CLI.txt: document proposed scp:-based CLI syntax
Brian Warner [Fri, 9 May 2008 19:36:19 +0000 (12:36 -0700)]
docs/CLI.txt: provide an overview of bin/tahoe subcommands
Brian Warner [Fri, 9 May 2008 01:02:55 +0000 (18:02 -0700)]
use a weakref cache in the client to manage singleton filenodes/dirnodes, to avoid autocollision. Should close #391.
Brian Warner [Fri, 9 May 2008 01:01:31 +0000 (18:01 -0700)]
check_memory: oops, fix HTTP-based modes: trailing slash in node.url vs no slash
Brian Warner [Thu, 8 May 2008 23:53:35 +0000 (16:53 -0700)]
test_dirnode: improve coverage of not-mutable-error a bit
Brian Warner [Thu, 8 May 2008 23:49:27 +0000 (16:49 -0700)]
check_memory: don't use fixed webports, they cause occasional test failures
Brian Warner [Thu, 8 May 2008 23:19:42 +0000 (16:19 -0700)]
deep-stats: add file-size histogram
Brian Warner [Thu, 8 May 2008 20:42:42 +0000 (13:42 -0700)]
test_dirnode: the size of the directories varies, tolerate it as low as 500 bytes
Brian Warner [Thu, 8 May 2008 20:33:07 +0000 (13:33 -0700)]
dirnode: refactor deep-stats a bit
Brian Warner [Thu, 8 May 2008 20:21:14 +0000 (13:21 -0700)]
dirnode: add a deep_stats(), like deep-size but with more information. webish adds t=deeps-size too.
Brian Warner [Thu, 8 May 2008 18:37:30 +0000 (11:37 -0700)]
run a stats provider even if there's no gatherer, since the HTTP /statistics page is then useful. Only run the once-per-second load-monitor if there is a gatherer configured
Brian Warner [Thu, 8 May 2008 01:36:37 +0000 (18:36 -0700)]
dirnode: use the concurrency limiter in t=manifest and t=deep-size, allow 10 retrievals in parallel
Brian Warner [Thu, 8 May 2008 01:35:53 +0000 (18:35 -0700)]
limiter: fix bug that ignores the limit= argument
robk-tahoe [Wed, 7 May 2008 23:42:20 +0000 (16:42 -0700)]
tahoefuse: fix typo which could cause data corruption
a typo in the 'flags2mode' code would wind up passing the O_APPEND
flag into the os open() call, which would cause the file to be opened
in 'strict append' mode, i.e. all writes extend the file, regardless of
calls to seek.
this causes a problem for tahoefuse in that the seek() calls made to
filehandles open through fuse would be ignored when write()s occurred.
this was evidenced by corruption seen when using rsync. it turns out
that rsync actually makes overlapping writes in some cases, i.e. even
when writing a new fresh file out, it still doesn't write a simple
contiguous span of data, but will make writes overlapping data already
written. this is probably related to the way it manages data blocks
internally for rolling checksums etc. at any rate, this bug would
thus cause rsync in those cases to write a chunk of duplicate data
into the file - leading to tahoe securely and reliably storing the
wrong data.
fixing this, so that non-append file opens do not pass O_APPEND seems
to eliminate this problem.
robk-tahoe [Wed, 7 May 2008 23:40:09 +0000 (16:40 -0700)]
tahoefuse: return bogus but useful data to statfs call
previously tahoefuse returned the fs stat for the filesystem the fuse plugin
was running upon (e.g. '/'). this works ok until you need to copy more to
tahoe than the local machine has free disk space, at which point Finder will
refuse to copy 'too much' data.
this changes it so that tahoe always reports 2TiB used of an 8TiB filesystem
this is entirely bogus, but allows copies of up to 2TiB to be initiated.
Brian Warner [Thu, 8 May 2008 00:00:47 +0000 (17:00 -0700)]
test_util: improve ConcurrencyLimiter tests
Brian Warner [Wed, 7 May 2008 23:53:30 +0000 (16:53 -0700)]
add a basic concurrency limiter utility
Brian Warner [Wed, 7 May 2008 23:52:48 +0000 (16:52 -0700)]
boringfile: add Twisted .egg files
Brian Warner [Wed, 7 May 2008 20:53:58 +0000 (13:53 -0700)]
web status: abbreviate file size in the upload+download summary page
Brian Warner [Wed, 7 May 2008 20:19:08 +0000 (13:19 -0700)]
munin: add tahoe_cpu_watcher.py, to track the data from misc/cpu-watcher.tac
Brian Warner [Wed, 7 May 2008 19:34:29 +0000 (12:34 -0700)]
misc/cpu-watcher*: add some tools to monitor CPU usage of arbitrary processes, like tahoe nodes
Zooko O'Whielacronx [Wed, 7 May 2008 15:39:03 +0000 (08:39 -0700)]
docs: fix a few stale comments in code
Zooko O'Whielacronx [Wed, 7 May 2008 12:37:11 +0000 (05:37 -0700)]
setup: tiny fix to syntax in makefile
Brian Warner [Tue, 6 May 2008 22:59:06 +0000 (15:59 -0700)]
configuration.txt: describe helper config
Zooko O'Whielacronx [Tue, 6 May 2008 22:29:04 +0000 (15:29 -0700)]
docs: shorter running.html
Brian Warner [Tue, 6 May 2008 20:49:01 +0000 (13:49 -0700)]
docs/helper.txt: explain more about the helper
Zooko O'Whielacronx [Tue, 6 May 2008 20:39:35 +0000 (13:39 -0700)]
docs: mention configuration, suggested by ben hyde's question about storage servers
Zooko O'Whielacronx [Tue, 6 May 2008 19:31:15 +0000 (12:31 -0700)]
docs: edit to install.html suggested by Brian
Zooko O'Whielacronx [Tue, 6 May 2008 19:30:56 +0000 (12:30 -0700)]
setup: trivial formatting change in _auto_deps.py
Zooko O'Whielacronx [Tue, 6 May 2008 19:09:00 +0000 (12:09 -0700)]
setup: remove specific checks for twisted dependency in makefile
Now that the twisted dependency is handled by the automatic dependency mechanism.
Zooko O'Whielacronx [Tue, 6 May 2008 18:17:47 +0000 (11:17 -0700)]
setup: require pycryptopp >= v0.5
Zooko O'Whielacronx [Tue, 6 May 2008 15:24:46 +0000 (08:24 -0700)]
setup: remove bundled pycryptopp-0.3.0.tar
Zooko O'Whielacronx [Tue, 6 May 2008 15:23:36 +0000 (08:23 -0700)]
setup: bundle pycryptopp-0.5.1.tar
pycryptopp-0.3.0 incorrectly crypts AES (depending on compiler and version of Crypto++ used)
Very soon now we're going to set up an "ext" repository to hold all tarballs and no longer check them into our trunk source tree.
Brian Warner [Wed, 30 Apr 2008 18:52:31 +0000 (11:52 -0700)]
test_stats.py: improve test coverage
Brian Warner [Wed, 30 Apr 2008 18:39:13 +0000 (11:39 -0700)]
stats: add tests for CPUUsageMonitor, modify it a bit to facilitate testing
Zooko O'Whielacronx [Wed, 30 Apr 2008 20:22:04 +0000 (13:22 -0700)]
setup: don't catch ImportError when importing _auto_deps in allmydata/__init__.py
Nowadays pkg_resources is a runtime requirement, and if there is something screwed up in the installation, we want an explicit ImportError exception as early as possible.
Brian Warner [Wed, 30 Apr 2008 01:24:57 +0000 (18:24 -0700)]
test_mutable: update notify_publish() to match new signature
Brian Warner [Wed, 30 Apr 2008 01:20:05 +0000 (18:20 -0700)]
mutable stats: track mutable bytes published too
Brian Warner [Wed, 30 Apr 2008 01:12:53 +0000 (18:12 -0700)]
stats: add CPU-percentage monitor, with 1min/5min/15min moving-window averages, using time.clock()
Zooko O'Whielacronx [Tue, 29 Apr 2008 22:51:58 +0000 (15:51 -0700)]
docs: remove the redundant (and therefore bit-rotting) parts of mutable-DSA.txt and instead refer to mutable.txt
Zooko O'Whielacronx [Tue, 29 Apr 2008 22:10:14 +0000 (15:10 -0700)]
wui: reorganize the welcome.xhtml page
Jake Edge tried Tahoe out and didn't notice the /status page. Hopefully with this new organization people like he will see that link more easily. This also addresses drewp's suggestion that the controls appear above the list of servers instead of below. (I think that was his suggestion.) I also reordered the controls.
Brian Warner [Mon, 28 Apr 2008 20:27:55 +0000 (13:27 -0700)]
offloaded: add 'resumes' counter to stats: how many upload requests are resuming an interrupted upload?
Zooko O'Whielacronx [Mon, 28 Apr 2008 14:05:44 +0000 (07:05 -0700)]
contrib: add a note about Armin Rigo's fuse implementation
ben [Mon, 28 Apr 2008 07:41:40 +0000 (00:41 -0700)]
support freebsd 6
Brian Warner [Thu, 24 Apr 2008 23:01:04 +0000 (16:01 -0700)]
debian: add python-setuptools to the debian install-time dependencies. Should close #382.
Brian Warner [Thu, 24 Apr 2008 18:30:38 +0000 (11:30 -0700)]
Makefile: add quicktest-figleaf: this is in my edit-test-repeat loop, and I need it to be fast
Brian Warner [Thu, 24 Apr 2008 18:28:35 +0000 (11:28 -0700)]
munin/tahoe_estimate_files.py: tool to estimate the total number of slots (mutable and immutable combined) in the grid, from a small sample
Zooko O'Whielacronx [Thu, 24 Apr 2008 17:29:17 +0000 (10:29 -0700)]
setup: pyOpenSSL is now easy_installable, and pycryptopp now includes Crypto++, so we can remove those two from the Manual Dependencies
robk-tahoe [Wed, 23 Apr 2008 22:50:29 +0000 (15:50 -0700)]
munin stats: fix typo in mutable file stats
robk-tahoe [Wed, 23 Apr 2008 21:39:41 +0000 (14:39 -0700)]
munin stats: add mutable/uploader traffic graphs
this adds munin graphs to present data already published by nodes to
the stats_gatherer, namely mutable files published/retrieved, and
immutable files uploaded, and the bytes thereof
Brian Warner [Wed, 23 Apr 2008 22:05:39 +0000 (15:05 -0700)]
introducer: only record one announcement per (tubid,service) tuple. Fixes #343.
Brian Warner [Wed, 23 Apr 2008 21:52:34 +0000 (14:52 -0700)]
client: don't start the IntroducerClient until the Tub is ready, otherwise we will sometimes connect to the introducer (or other clients) before we've done Tub.setLocation, which loses some information on the introducer status page
Brian Warner [Wed, 23 Apr 2008 21:03:23 +0000 (14:03 -0700)]
munin/tahoe-stats.py: category is 'counters' not 'stats', use DERIVE
Brian Warner [Wed, 23 Apr 2008 01:53:20 +0000 (18:53 -0700)]
test_mutable: test that all servers refusing our share means a publish fails
Brian Warner [Wed, 23 Apr 2008 00:25:14 +0000 (17:25 -0700)]
mutable: improve test coverage in Retrieve, when shares change after mapupdate
Brian Warner [Wed, 23 Apr 2008 00:14:26 +0000 (17:14 -0700)]
hush pyflakes warning about code that got moved in the recent StallMixin refactoring
Brian Warner [Tue, 22 Apr 2008 23:47:52 +0000 (16:47 -0700)]
mutable/servermap: improve test coverage
Brian Warner [Tue, 22 Apr 2008 23:47:15 +0000 (16:47 -0700)]
testutil: factor stall() out into a common location
Brian Warner [Tue, 22 Apr 2008 19:54:16 +0000 (12:54 -0700)]
test_introducer.py: don't log nodeids as binary goop
Brian Warner [Tue, 22 Apr 2008 18:51:14 +0000 (11:51 -0700)]
web/status: remove trailing whitespace
Brian Warner [Tue, 22 Apr 2008 18:49:53 +0000 (11:49 -0700)]
mutable: test write failures, uncoordinated write detection
Brian Warner [Tue, 22 Apr 2008 00:51:22 +0000 (17:51 -0700)]
mutable: remove some dead code, rearrange use of populate_pubkey
Brian Warner [Tue, 22 Apr 2008 00:50:50 +0000 (17:50 -0700)]
mutable: improve testing to exercise the non-cached case
Brian Warner [Tue, 22 Apr 2008 00:29:59 +0000 (17:29 -0700)]
check_speed: rearrange a bit to make it easy to disable certain portions
Brian Warner [Tue, 22 Apr 2008 00:29:12 +0000 (17:29 -0700)]
mutable status: make the 'total' timing value optional: unit tests fail otherwise
Brian Warner [Tue, 22 Apr 2008 00:27:50 +0000 (17:27 -0700)]
mutable read: enable the cache (written during mapupdate, read during retrieve). This speeds up small-file reads by about 30% over a link with an average 25ms RTT
Brian Warner [Mon, 21 Apr 2008 23:16:55 +0000 (16:16 -0700)]
mutable status: use google-chart-API to draw server response times for servermap update
Brian Warner [Mon, 21 Apr 2008 22:10:50 +0000 (15:10 -0700)]
mutable: improve test coverage slightly
Brian Warner [Mon, 21 Apr 2008 19:19:17 +0000 (12:19 -0700)]
download status: add time spent paused by the client (when we're serving over a slow HTTP link)
Zooko O'Whielacronx [Thu, 24 Apr 2008 16:57:04 +0000 (09:57 -0700)]
setup: test depends on build, which means it invokes setup.py every time, which is slower but does "the right thing" more often
There is a new target "quicktest" which depends on the .built and .checked-deps files. test-figleaf also depends on the build target now.
Zooko O'Whielacronx [Wed, 23 Apr 2008 20:41:56 +0000 (13:41 -0700)]
setup: remove the misc/hatch-eggs.py script, which we no longer use since [1937]
Peter Secor [Tue, 22 Apr 2008 00:38:04 +0000 (17:38 -0700)]
windows installer - removing uninstall tests, trying to get to a clean build that works as it's not clear that this latest build can update the virtual drive properly
Peter Secor [Mon, 21 Apr 2008 23:41:43 +0000 (16:41 -0700)]
windows installer - added -Q option to installer as well as to uninstaller, added back the SMB service stop
Peter Secor [Mon, 21 Apr 2008 23:18:29 +0000 (16:18 -0700)]
windows installer - added line to quit the system tray executable which automatically stops the SMB service, removed lnow redundant line to stop SMB service
Zooko O'Whielacronx [Tue, 22 Apr 2008 19:28:18 +0000 (12:28 -0700)]
key_generator: make default key size be a constructor argument instead of a class variable, pass default key size of 522 (the smallest that we can do) in unit tests to make them faster
Zooko O'Whielacronx [Tue, 22 Apr 2008 19:07:12 +0000 (12:07 -0700)]
setup: remove obsolete makefile target build-deps
Zooko O'Whielacronx [Mon, 21 Apr 2008 13:37:13 +0000 (06:37 -0700)]
setup: windows: make sub-ver.py treat None as 0
Brian Warner [Sat, 19 Apr 2008 03:13:28 +0000 (20:13 -0700)]
mutable/servermap: improve test coverage
Brian Warner [Sat, 19 Apr 2008 02:55:12 +0000 (19:55 -0700)]
mutable: improve test coverage, fix bug in privkey fetching, add .finished to stats, remove dead code
Zooko O'Whielacronx [Fri, 18 Apr 2008 20:24:59 +0000 (13:24 -0700)]
setup: remove the try: except: around the import of pkg_resources -- we now require setuptools at run time and at build time
Brian Warner [Fri, 18 Apr 2008 07:43:29 +0000 (00:43 -0700)]
mutable.node: avoid reentrancy problems in Deferred code on twisted2.5, by adding an eventual-send call
Zooko O'Whielacronx [Fri, 18 Apr 2008 19:17:22 +0000 (12:17 -0700)]
setup: don't try __import__(name) in _auto_deps.py
This happens to work, because all of our "distribution" (i.e. distributable packaged Python code) names to coincide with all of their "package" (i.e. a directory with a __init__.py in it, which is "import"-able) names, except, I think for Twisted on Brian's debian sid system.
But there's no reason why it should always work, and the only reason for that __import__() was to give us an explicit error message indicating missing requirements in the case that pkg_resources isn't importable or that the requirements don't have correct .egg-info metadata. So, by removing this stanza we may allow certain places to get a more ad-hoc failure message, i.e. an ImportError from somewhere, instead of an ImportError from _auto_deps.py, but that's okay.
Note that dependencies which do not have their .egg-info metadata with them are increasingly rare, since Python 2.5 distutils creates the .egg-info file by default, and Linux distributions have stopped their former practice of actively deleting the .egg-info files.
Zooko O'Whielacronx [Fri, 18 Apr 2008 18:58:00 +0000 (11:58 -0700)]
setup: trivial change to metadata to test patch/branch/build system
Zooko O'Whielacronx [Fri, 18 Apr 2008 04:32:38 +0000 (21:32 -0700)]
let MutableFileNode.__repr__() return something useful even if it isn't not initialize yet
This is because I sometimes log or examine a repr of an object from a function which is being called from that object's own __init__()...
(I'm committing this patch in order to test our patch management infrastructure.)
Zooko O'Whielacronx [Fri, 18 Apr 2008 04:09:12 +0000 (21:09 -0700)]
docs: tiny update to webapi.txt
I'm actually committing this just to test our patch management infrastructure.
Zooko O'Whielacronx [Fri, 18 Apr 2008 04:00:51 +0000 (21:00 -0700)]
docs: tiny change in webapi.txt
(I'm actually committing this patch only in order to test our patch management infrastructure.)
Zooko O'Whielacronx [Fri, 18 Apr 2008 03:57:41 +0000 (20:57 -0700)]
docs: merge conflicts between the patch to document "127.0.0.1" instead of "localhost" and some other patches (precisely which, I don't know)
Zooko O'Whielacronx [Fri, 18 Apr 2008 03:45:34 +0000 (20:45 -0700)]
docs: use "127.0.0.1" instead of "localhost"
Unfortunately there are occasionally configurations in the real world where "localhost" does not resolve to 127.0.0.1, and if a user has such a configuration then using 'localhost' could lead to an authority leak.
Zooko O'Whielacronx [Fri, 18 Apr 2008 05:07:52 +0000 (22:07 -0700)]
setup: a tiny tweak to setup to avoid requiring darcsver package if the user isn't invoking "./setup.py darcsver"
The real reason for this patch is to test our patch management infrastructure.
Brian Warner [Fri, 18 Apr 2008 03:06:06 +0000 (20:06 -0700)]
dirnode: return to 'delete fails if the child wasn't actually there' semantics, to make tests pass. There's a switch to enable/disable this
Brian Warner [Fri, 18 Apr 2008 02:57:55 +0000 (19:57 -0700)]
test_mutable: hush pyflakes
Brian Warner [Fri, 18 Apr 2008 02:57:04 +0000 (19:57 -0700)]
dirnode: update to use MutableFileNode.modify
Brian Warner [Fri, 18 Apr 2008 02:56:45 +0000 (19:56 -0700)]
testutil: minor reformatting
Brian Warner [Fri, 18 Apr 2008 02:55:51 +0000 (19:55 -0700)]
test_mutable: factor out ShouldFailMixin
Brian Warner [Fri, 18 Apr 2008 02:12:42 +0000 (19:12 -0700)]
mutable: implement MutableFileNode.modify, plus tests
Brian Warner [Fri, 18 Apr 2008 00:51:38 +0000 (17:51 -0700)]
mutable: replace MutableFileNode API, update tests. Changed all callers to use overwrite(), but that will change soon
Brian Warner [Thu, 17 Apr 2008 20:09:22 +0000 (13:09 -0700)]
mutable WIP: merge in patches from current trunk