Renamings (snake_case instead of camelCase; follow naming conventions).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Simplifications.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Cosmetics.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add a test that a tree moved into the drop-upload directory is watched.
(It currently hangs because this case isn't handled correctly.)

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Refactor test_move_tree and make directory names non-ASCII.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Simplify test skipping.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Remove debug print statements

Teach uploader to only unicode instead of utf-8

Fixed some tests

Unicode fixes; also remove some duplication in test methods.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Rename confusing method name.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Move initialization to setUp().

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Unfix tests :-p

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Fix tests :-D

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Cosmetics (ordering of checks).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

drop-upload: make paths consistently Unicode. Also fix import warnings.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

test_drop_upload.py: factor out duplicated setting of self.inotify.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

fix dir tree unit test

Fix _test_persistence AND remove superfluous print statements

Fix problems with _test_db_persistence unit test code

Create another uploader db test

Remove unneeded lines the crept in from previous WIP commits

Cosmetics.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Avoid shadowing a method name.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Better variable name.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Change when _scan is called.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

For persistence test, use a new client (with the same backupdb).

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Simplification of _upload_lazy_tail.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

WIP

wip - confirming that the db persistence doesn't work

Fix db persistence unit test for uploader

Fix race condition in upload deque

Remove redundant calls to _scan and startService.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

WIP

Fixed move tree test and cleaned up persistence test

- the persistence test does not yet work

Use Unicode directory name.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add temporary debug prints.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Whitespace.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Try to fix potential race conditions in testMoveEmptyTree.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add ignore mechanism to upload callback.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Clarify dir/file/link logic.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add recursive=True to drop_upload watch.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

WIP
- change the upload deque to handle file path strings instead of
FilePath objects

fix unit tests WIP

Add stats counter checking to uploader service start test

Add further refinements to move tree uploader test
- test empty tree and small tree test cases using various stats counters

Uncomment mkdir in uploader unit test

Add move empty dir tree test and minor test cleanup

Remove code duplication from uploader unit tests

Don't increment disappeared stats twice

Merge branch '2406.otf-objective-2.3.1-subdirs' into 2406.otf-objective-2.3.1-fix-upload-deque

Teach uploader to monitor subdirectories

Teach uploader to keep object stats not file stats

- we keep track of objects instead of files so that we are not
forced to determine the object type (file, directory, symlink, special file etc.)

Fix test_create_drop_uploader

This test must use the correct function signature for it's mock function...
obviously. =-p

Remove filepath from pending set earlier

Fix test_create_drop_uploader

This test must use the correct function signature for it's mock function...
obviously. =-p

Teach upload deque to be sequential and asynchronous

Teach uploade deque append to not block (potentially?)

Simply upload queue API

Merge branch 'dropupload-scan-on-startup-1458-1' into 2406.otf-objective-2.3.1

Conflicts:
src/allmydata/frontends/drop_upload.py
src/allmydata/test/test_drop_upload.py

Make the unit tests work for Mock and Real

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add very basic unit tests for dir scan and db

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Scan dir when service starts

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

WIP - add some backup db code fragments

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Bug fix: don't append filepath to upload deque twice

Bug fix: deque must be unbounded

If using a bounded deque then:
"""
Once a bounded length deque is full, when new items are added, a corresponding number of items are discarded from the opposite end.
"""

This bug was introduced at commit 87657eb382da05039cce99862691efcc27243193

Fix pending upload bug

- remove filepath from pending upload set BEFORE performing upload
This bug was introduced at commit 87657eb382da05039cce99862691efcc27243193

Disable a test that breaks on Windows.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

WIP: workaround for duplicate events on Windows. refs #2220

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Windows support for drop-upload. fixes #1431

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Docs for drop-upload on Windows.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

DropUploader.startService returns a deferred; don't drop it.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Use explicit deque to create expected upload "queue" functionality.
Deduplicate upload events via pending set.
refs #1440, #1449

Author: David Stainton <dstainton415@gmail.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Teach StorageFarmBroker to fire a deferred when a connection threshold is reached. refs #1449

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

backupdb.py: cosmetics

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Make backupdb use dbutil. (This also causes it to use "PRAGMA synchronous = OFF".)

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add util/dbutil.py: open/create/update sqlite databases given some schema.

Author: Brian Warner <warner@lothar.com>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Some useful Deferred utilities, originally from the cloud backend branch.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Make the unit tests work for Mock and Real

Add very basic unit tests for dir scan and db

Scan dir when service starts

the add file to queue function currently does nothing...

Merge pull request #152 from vu3rdd/182-osx-packaging-7

strip the single quotes around $VERSION

strip the single quotes around $VERSION

Without stripping the single quotes, the generated pagkage filename
also has the undesired single quotes around the version string.
(eg: tahoe-lafs-'1.10.0.post309'-osx.pkg).

WIP - add some backup db code fragments

README.rst: change description to the new wording on tahoe-lafs.org (with "file store" instead of "filesystem").

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add discussion and rejection of Change Journals.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add link to otf-magic-folder-objective ticket query.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Copyediting.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Collaborative design from Daira and David.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Merge branch 'ftp-2394'

closes ticket:2394

typo

bump Twisted dep to 11.1.0, thus simplify IntishPermissions

test_ftp: fix comparison against integer

Fix ftp 'ls' to work with current Twisted-15.0.0

refs ticket:2394

It's kind of a hack, but Twisted changed the API and I couldn't find a
cleaner way to detect which form of "permissions" value the Twisted FTP
server wants.

I've manually tested it against 14.0.2 and 15.0.0.

Merge pull request #149 from warner/mutable-2034

use more servers for test_mutable.Update, to test #2034 properly

use more servers for test_mutable.Update, to test #2034 properly

Add a test, add missing imports. refs #2388

This tests ftpd, but not sftpd. Doing this sort of test on sftpd
requires the creation of a valid pubkey/privkey file pair, which is more
work than I want to do right now.

init_ftp/init_sftp were changed to interpret the configured
accounts.file as relative to the node's basedir, with
abspath_expanduser_unicode(accountfile, base=self.basedir).
This would happen naturally in a real node, since it os.chdir()s
to the basedir before doing anything. But tests don't do that.

Author: Brian Warner <warner@lothar.com>
Author: Daira Hopwood <daira@jacaranda.org>
Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Ensure that path parameters to SFTPServer and FTPServer constructors are unicode. refs #2388

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add from_utf8_or_none and tests.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Fix a missing import in test-osx-pkg.py. refs #2393

Signed-off-by: Daira Hopwood <daira@jacaranda.org>