Capture exceptions from conflict and write info in magic folder db

Fix _write_downloaded_file by adding base path

Correct spelling is os.utime not utimes

Add Downloader._write_downloaded_file. refs ticket:2489

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Remove debug prints in backupdb.py.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Remove or comment out print statements.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Rename Client.upload_ready_d to connected_enough_d.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

In Downloader._get_collective_latest_file, consume errors when getting metadata.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Teach uploader to process lazy tail upon stop method

Teach magic-folder ready method to return a deferred

Fix a bug in Downloader._scan_remote that incorrectly skips subsequent files
(and drops a Deferred) when local version >= remote version.
Also simplify that code and robustify it for the case where the version metadata is missing.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Use deferredutil.HookMixin to simplify callback for tests.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Minor fix to alice bob test

Fix _scan_remote and remove print statements

Fix incorrect order of fields in SQL statement.
My kingdom for a better SQL DSL.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Cleanup.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

More debug prints.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Debug prints to diagnose version mismatch after deletion (backupdb).

Debug prints to diagnose version mismatch after deletion.

Add some print statements.

Merge pull request #184 from vu3rdd/2477.refactor-all-the-things.0

test for get_pathinfo()

test_get_pathinfo: testdir is unnecessary

Use basedir itself, instead of creating another directory inside basedir.

test_get_pathinfo: rename the identifier d to testdir

d, by convention, is used for the Twisted Deferred objects.

add more tests for get_pathinfo()

When operating on a file, test if the "file" is not a directory
or a symlink. Likewise for directory and symlink.

Add test for get_pathinfo()

Use get_pathinfo instead of separate os.path.{is*,exists} calls.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Extend get_pathinfo to add size, ctime and mtime.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Remove an unnecessary precondition and an incorrect comment.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add fileutil.get_pathinfo function.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Extract get_metadata and get_filenode to methods of Uploader.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Processing symlinks and special files should not be an error.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Refactor uploader by inlining _add_file into _process._maybe_upload.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Refactor uploader by inlining _add_file into _process._maybe_upload.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Refactor uploader by extracting _process_child to a top-level method.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Swap two checks to clarify which is failing.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

fix code that adds version number of deleted file to magic-folder db

enable version checking tests - currently a righteous fail

make various changes - work in progress

add version checking to alice bob test

Update magic-folder db after download

Add final download to alice bob test

- that is to say after alice deletes the file she rewrites it
and this causes bob to download this latest version

Remove old unused append to deque function from downloader

WIP to unify queue processing between uploader and downloader.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Repair tests.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Move uploader's _append_to_deque and _turn_deque to QueueMixin.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Make sure that counter names and log messages include "uploader" or "downloader".

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Trivial fix.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

remove debugging print statements

Fix alice bob test

Teach downloader to stop when told to stop

Add logic to only add object to download queue when appropriate

wip

Try to fix some tests - WIP.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Rename and refactor path variables to be more consistent.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

WIP - fix some bugs

Refactor MagicFolder into Uploader and Downloader classes (WIP). refs ticket:2477

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Simplify and document the type of MagicFolder._download_scan_batch.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Remove redundant call to abspath_expanduser_unicode.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Reduce code duplication and make sure errors from delegate commands are output.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Change INVITE_SEPARATOR to '+'.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Rename MagicFolderCLITestMixin to MagicFolderTestMixin.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Fix tests.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Add checks (that currently fail) for version in local db.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Use relative paths in magic folder db.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

WIP

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Fix some pyflakes errors and warnings.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

Repair backupdb.

Signed-off-by: Daira Hopwood <daira@jacaranda.org>

fix bug in test_errors and add test

- test that we report the correct error when
a write cap is passed as a collective dircap

add kw arg to _get_count for tests

make test_magic_folder use non-ascii chars in nickname

Make alice bob test use non-ascii chars in nicknames

add comments and blank lines to make setup_alice_and_bob more readable

rename to do_invite and do_join for tests

rename to do_create_magic_folder for tests

revert previous additional blank line

wrap long import line

wrap long line and spell magic folder without hyphen

Spell magic folder without the hyphen

wrap long line

replace bare asserts with usage error message

get rid of another diminish hack and force alias args to end with ':'

get rid of terrible diminish cap hack

minor corrections

slightly better error message

change spelling module variable INVITE_SEPARATOR

don't double increment version

start with version 0

simplify _db_file_is_uploaded

better handle download failures

remove superfluous plus sign

fix remote scan procedure

replace calls to filter with list comprehension

simplify _should_download helper method

rename ignore count to upload ignore count

remove superfluous constructor which the parent class already has

Make backupdb schema evolution more visible

alice bob test alice rewrite file

unlink old version of file from remote tahoe upload dirnode

for local fs scan compare local version with remote

Fix test_errors to check for exception message associated with
an invalid collective dircap.