From: Kevan Carstensen Date: Mon, 24 May 2010 00:35:08 +0000 (-0700) Subject: Add a specification for servers of happiness. X-Git-Tag: trac-4500~18 X-Git-Url: https://git.rkrishnan.org/%5B/%5D%20/uri/reliability?a=commitdiff_plain;h=7cadb49b88c03209e357b66727d22677001141a3;p=tahoe-lafs%2Ftahoe-lafs.git Add a specification for servers of happiness. --- diff --git a/docs/specifications/servers-of-happiness.txt b/docs/specifications/servers-of-happiness.txt new file mode 100644 index 00000000..493094bf --- /dev/null +++ b/docs/specifications/servers-of-happiness.txt @@ -0,0 +1,86 @@ += Servers of Happiness = + +When you upload a file to a Tahoe-LAFS grid, you expect that it will +stay there for a while, and that it will do so even if a few of the +peers on the grid stop working, or if something else goes wrong. An +upload health metric helps to make sure that this actually happens. An +upload health metric is essentially a test that looks at a file on a +Tahoe-LAFS grid and says whether or not that file is healthy; that is, +whether it is distributed on the grid in such a way as to ensure that it +will probably survive in good enough shape to be recoverable even if a +few things go wrong between the time of the test and the time that it is +recovered. Our current upload health metric for immutable files is called +'servers-of-happiness'; its predecessor was called 'shares-of-happiness'. + +shares-of-happiness used the number of encoded shares generated by a +file upload to say whether or not it was healthy. If there were more +shares than a user-configurable threshold, the file was reported to be +healthy; otherwise, it was reported to be unhealthy. In normal +situations, the upload process would distribute shares fairly evenly +over the peers in the grid, and in that case shares-of-happiness +worked fine. However, because it only considered the number of shares, +and not where they were on the grid, it could not detect situations +where a file was unhealthy because most or all of the shares generated +from the file were stored on one or two peers. + +servers-of-happiness addresses this by extending the share-focused +upload health metric to also consider the location of the shares on +grid. servers-of-happiness looks at the mapping of peers to the shares +that they hold, and compares the cardinality of the largest happy subset +of those with a user-configurable threshold (A happy subset of peers has +the property that any k (where k is as in k-of-n encoding) peers within +the subset can reconstruct the source file). This definition of file +health provides a stronger assurance of file availability over time; +with 3-of-10 encoding, and happy=7, a healthy file is still guaranteed +to be available even if 4 peers fail. + +== Measuring Servers of Happiness == + +We calculate servers-of-happiness by computing a matching on a +bipartite graph that is related to the layout of shares on the grid. +One set of vertices is the peers on the grid, and one set of vertices is +the shares. An edge connects a peer and a share if the peer will (or +does, for existing shares) hold the share. The size of the maximum +matching on this graph is the size of the largest happy peer set that +exists for the upload. + +First, note that a bipartite matching of size n corresponds to a happy +subset of size n. This is because a bipartite matching of size n implies +that there are n peers such that each peer holds a share that no other +peer holds. Then any k of those peers collectively hold k distinct +shares, and can restore the file. + +A bipartite matching of size n is not necessary for a happy subset of +size n, however (so it is not correct to say that the size of the +maximum matching on this graph is the size of the largest happy subset +of peers that exists for the upload). For example, consider a file with +k = 3, and suppose that each peer has all three of those pieces. Then, +since any peer from the original upload can restore the file, if there +are 10 peers holding shares, and the happiness threshold is 7, the +upload should be declared happy, because there is a happy subset of size +10, and 10 > 7. However, since a maximum matching on the bipartite graph +related to this layout has only 3 edges, Tahoe-LAFS declares the upload +unhealthy. Though it is not unhealthy, a share layout like this example +is inefficient; for k = 3, and if there are n peers, it corresponds to +an expansion factor of 10x. Layouts that are declared healthy by the +bipartite graph matching approach have the property that they correspond +to uploads that are either already relatively efficient in their +utilization of space, or can be made to be so by deleting shares, and +that place all of the shares that they generate, enabling redistribution +of shares later without having to re-encode the file. Also, it is +computationally reasonable to compute a maximum matching in a bipartite +graph, and there are well-studied algorithms to do that. + +== Issues == + +The uploader is good at detecting unhealthy upload layouts, but it +doesn't always know how to make an unhealthy upload into a healthy +upload if it is possible to do so; it attempts to redistribute shares to +achieve happiness, but only in certain circumstances. The redistribution +algorithm isn't optimal, either, so even in these cases it will not +always find a happy layout if one can be arrived at through +redistribution. We are investigating improvements to address these +issues. + +We don't use servers-of-happiness for mutable files yet; this fix will +likely come in Tahoe-LAFS version 1.8.