Saturday, May 15, 2010

Wikitrends 2.0

Wikitrends 2.0 is now live. Rewrote and optimized most of it. Compared to the old version a single update now takes 1/10 of the time, with 10x more data. The site now has Wikipedia page views trends over three different time periods; montly, weekly and daily (last version only did daily.) I also moved the site to Wikimedia Toolserver. Thanks to Wikimedia Deutschland for hosting it on their two monster 8-core machines.

The monthly trends, which updates once per day, are based on 432 GB of uncompressed data. To make it even remotely possible to work with without a huge cluster of machines, I filter, aggregate and compress it down to a mere 10 GB.

No comments: