Four Changes which Reset Recency Metric for Sort Order

As was mentioned at eBay Live in one of the seller panels: if a seller changes any of these it creates a resetting of the recency metric for preference in sort order:

Title
Category
Price Increase
Item Condition

I have not had a chance to test this in any way, but it seemed quite interesting to me at the time.

Incremental Changes vs. Large Changes to UI

What is the correct level of granularity for changes to a user interface or policy? A price increase to subscription fees may best be done slowly, and a price decrease done quickly to make it more dramatic. But a UI change can have significant and startling effects. Users habituate their UI behavior. A link with a name change, such as having a link to a subsite with automotive content in 12 point font change to a link for an express general sales format, may divert folks unexpectedly who have habituated clicking on the automotive link.

I saw a site which made a change so that it moved it's "donate money" box and button to the part of the page where the mouse would be on the previous page clicked through. Remarkably effective, I did donate! The box was moved directly, not incrementally.

Users sometimes complain about incremental changes when they can't aggregate because of it. An example is policy changes every few months requiring changes to submitted content. If a change requires a change to user behavior to accommodate it, such as needing to remove email addresses from pages, and then a later policy change to require elimination of JS, then larger, infrequent changes may be more appropriate.

Mean or Median?

Sometimes data is so noisy that the mean may vary significantly because of outliers. Although the use of median loses some of the data from mean, if the ratio of mean to median remains relatively stable over time (or within a scaled index) is median a valid metric? Generally a company wants to improve total (and mean) metrics for something like revenue. The danger of moving median solely is that the outliers (which may provide significant upside), are lost or deemphasized.

Merging and Splitting Social Networks

Digital social networks and communities have the same property as the stock market - in the words of Alan Greenspan "They will fluctuate." One challenge of managing these networks is managing the connectivity as they grow or shrink. If the network is too large and/or too broad, the relevance of the network to the participants may be hindered. Creating "shards" - smaller communities where people can recognize each other reliably, is one technique. Local trading communities like the city sites of Craigslist are essentially "shards," verticalizing one aspect of the users. In this case "cities" are a type of category and each site is essentially a "local" category. World of Warcraft is a different type of "shard." Although the players are relatively homogenous, the population is arbitrarily by default separated and placed in distinct servers, with no ability to trade virtual goods between servers. Other marketplaces like Amazon do no sharding whatsoever, preferring to keep the inventory universally displayed and available for all users.

There is a technology benefit to sharding - managing scalability is much easier. With a population of 10,000,000 active users and about 100 servers, WoW has 100,000 users or so in each of its shards. Perhaps a few hundred or thousand are active at any time, keeping the connectivity scalable for a single server (or a few/backup servers). Since the servers are independent any failure or crash is localized. Additionally, Blizzard/Activision can deploy new code on a few servers instead of universally, and examine both the technical and community impact on a microcosm scale.

Amazon's changes, on the other hand, may affect the entire site and community inadvertently. Even a test where 1% of users are exposed to a function can have a huge effect. If that 1% happens to include a large seller who is now in a test group with a functionality that allows spamming of listings, or undercutting of prices instantly, that one account can quickly dominate the entire community. Shards allow containment, universal marketplaces do not.

Another technology challenge is scale. In essence a completely connected community requires the bandwidth of a full crossbar. Even the attempts at heuristics like a benes network have worst case complexity of n^2 where n = the number of users. For s = number of shards, compare that to the (s * (n/s)^2) scalability of sharded communities. Assume Amazon and WoW have 10M monthly active users. If we just consider that 1% of these users may be active at a given time, Amazon requires a non-blocking full crossbar of scale 100,000^2 = 10,000,000,000, whereas Blizzard needs 100 full crossbars of size 1,000^2. That's just 100 x 1,000,000 = 100,000,000 total. Blizzard (and by extension Craigslist) needs crossbar connectivity that is 100 times less. This may partially explain why Craigslist has dozens of employees and Amazon has thousands.

The drawbacks of shards may be an issue however. In the case of WoW, it may be difficult to find another player who wants to group for a certain objective, while across other servers there are many other players wanting to play with you. Blizzard has designed special servers, essentially hybrids, where players can group (but NOT trade virtual goods). In this hybrid model players are still able to communicate with their "home" server while playing, but are accessing a special "group" server for the most bandwidth for a few minutes to an hour. The hybrid temporary community on it's own server ensures the wait to find other players for the group is not excessive.

The sharded marketplace, Craigslist, does not have a way to search across communities by default. If a good is fairly rare, it may not exist on one particular city site, and the user needs to search on all the different sites. There is no functionality to do a cross-city search.