Merging and Splitting Social Networks

Digital social networks and communities have the same property as the stock market - in the words of Alan Greenspan "They will fluctuate." One challenge of managing these networks is managing the connectivity as they grow or shrink. If the network is too large and/or too broad, the relevance of the network to the participants may be hindered. Creating "shards" - smaller communities where people can recognize each other reliably, is one technique. Local trading communities like the city sites of Craigslist are essentially "shards," verticalizing one aspect of the users. In this case "cities" are a type of category and each site is essentially a "local" category. World of Warcraft is a different type of "shard." Although the players are relatively homogenous, the population is arbitrarily by default separated and placed in distinct servers, with no ability to trade virtual goods between servers. Other marketplaces like Amazon do no sharding whatsoever, preferring to keep the inventory universally displayed and available for all users.

There is a technology benefit to sharding - managing scalability is much easier. With a population of 10,000,000 active users and about 100 servers, WoW has 100,000 users or so in each of its shards. Perhaps a few hundred or thousand are active at any time, keeping the connectivity scalable for a single server (or a few/backup servers). Since the servers are independent any failure or crash is localized. Additionally, Blizzard/Activision can deploy new code on a few servers instead of universally, and examine both the technical and community impact on a microcosm scale.

Amazon's changes, on the other hand, may affect the entire site and community inadvertently. Even a test where 1% of users are exposed to a function can have a huge effect. If that 1% happens to include a large seller who is now in a test group with a functionality that allows spamming of listings, or undercutting of prices instantly, that one account can quickly dominate the entire community. Shards allow containment, universal marketplaces do not.

Another technology challenge is scale. In essence a completely connected community requires the bandwidth of a full crossbar. Even the attempts at heuristics like a benes network have worst case complexity of n^2 where n = the number of users. For s = number of shards, compare that to the (s * (n/s)^2) scalability of sharded communities. Assume Amazon and WoW have 10M monthly active users. If we just consider that 1% of these users may be active at a given time, Amazon requires a non-blocking full crossbar of scale 100,000^2 = 10,000,000,000, whereas Blizzard needs 100 full crossbars of size 1,000^2. That's just 100 x 1,000,000 = 100,000,000 total. Blizzard (and by extension Craigslist) needs crossbar connectivity that is 100 times less. This may partially explain why Craigslist has dozens of employees and Amazon has thousands.

The drawbacks of shards may be an issue however. In the case of WoW, it may be difficult to find another player who wants to group for a certain objective, while across other servers there are many other players wanting to play with you. Blizzard has designed special servers, essentially hybrids, where players can group (but NOT trade virtual goods). In this hybrid model players are still able to communicate with their "home" server while playing, but are accessing a special "group" server for the most bandwidth for a few minutes to an hour. The hybrid temporary community on it's own server ensures the wait to find other players for the group is not excessive.

The sharded marketplace, Craigslist, does not have a way to search across communities by default. If a good is fairly rare, it may not exist on one particular city site, and the user needs to search on all the different sites. There is no functionality to do a cross-city search.