4 This tool estimates how much space would be consumed by a filetree into which
5 a native directory was copied.
7 One open question is how we should encode directories. One approach is to put
8 a block of data on a server, one per directory, which effectively contains a
9 dictionary that maps child names to targets (URIs for children which are
10 files, slotnames for children which are directories). To prevent the server
11 which hosts this data from either learning its contents or corrupting them,
12 we can add encryption and integrity checks to the data, at the cost of
15 This program is intended to estimate the size of these data blocks using
16 real-world filenames and directories. You point it at a real directory, and
17 it does a recursive walk of the filesystem, adding up the size of the
18 filetree data structures that would be required to represent it.
22 A: no confidentiality or integrity checking. Directories are serialized
23 plaintext dictionaries which map file/subdir names to targets (either
24 URIs or slotnames). Each entry can be changed independently.
25 B1: child names and targets are encrypted. No integrity checks, so the
26 server can still corrupt the contents undetectably. Each entry can
27 still be changed independently.
28 B2: same security properties as B1, but the dictionary is serialized before
29 encryption. This reduces overhead at the cost of preventing independent
30 updates of entries (all entries must be updated at the same time, so
31 test-and-set operations are required to avoid data-losing races)
32 C1: like B1, but adding HMACs to each entry to guarantee data integrity
33 C2: like B2, but adding a single block-wide HMAC for data integrity
39 #URI:7jzbza6iwdsk5xbxsvdgjaugyrhetw64zpflp4gihmyh5krjblra====:a5qdejwbimu5b2wfke7xwexxlq======:gzeub5v42rjbgd7ccawnahu2evqd42lpdpzd447c6zkmdvjkpowq====:25:100:219889
40 # that's a printable representation of two 32-byte hashes (storage index, URI
41 # extension block hash) and a 16-byte AES read-capability key, and some
42 # share-count and size information
45 #pb://xextf3eap44o3wi27mf7ehiur6wvhzr6@207.7.153.180:56677,127.0.0.1:56677/zilcw5uz2yyyo===
46 # that's a FURL which points at the slot. Modes that need to add a
47 # read-capability AES key will need more space.
51 def slotsize(mode, numfiles, numdirs):
52 # URI_sizes is the total space taken up by the target (dict keys) strings
53 # for all of the targets that are files, instead of directories
54 target_sizes_for_files = numfiles * URI_SIZE
55 slotname_size = SLOTNAME_SIZE
56 if mode in ("B1", "B2", "C1", "C2"):
58 # slotname_sizes is the total space taken up by the target strings for
59 # all the targets that are directories, instead of files. These are
60 # bigger when the read+write-cap slotname is larger than the store-cap,
61 # which happens as soon as we seek to prevent the slot's host from
62 # reading or corrupting it.
63 target_sizes_for_subdirs = numdirs * slotname_size
65 # now how much overhead is there for each entry?
66 per_slot, per_entry = 0, 0
70 per_entry = 16+12+12 + 32+32
75 num_entries = numfiles + numdirs
76 total = (target_sizes_for_files +
77 target_sizes_for_subdirs +
79 per_entry * num_entries
83 MODES = ("A", "B1", "B2", "C1", "C2")
86 total = dict([(mode,0) for mode in MODES])
89 for absroot, dirs, files in os.walk(root):
91 #print " %d files" % len(files)
92 #print " %d subdirs" % len(dirs)
93 num_files += len(files)
95 stringsize = len(''.join(files) + ''.join(dirs))
97 total[mode] += slotsize(mode, len(files), len(dirs)) + stringsize
99 print "%d directories" % num_dirs
100 print "%d files" % num_files
101 for mode in sorted(total.keys()):
102 print "%s: %d bytes" % (mode, total[mode])
105 if __name__ == '__main__':
109 260:warner@monolith% ./count_dirs.py ~