[Utopian-IO] Retrieving Top Projects with the Utopian API

The backend has new API endpoints for retrieving the top projects on the Utopian. This can be used for statistical or analytical purposes or for a front end to display the top projects. Other bug fixes were made as well :)

Here's what can be be done out of the box

Endpoint to retrieve the top 3 projects based on number of contributions in the last week
/api/posts/top?limit=3&start_date=2017-12-1&end_date=2017-12-8

Endpoint to retrieve the top new 4 projects of the last week based on the number of contributions/rewards
/api/posts/top?limit=4&start_date=2017-12-1&end_date=2017-12-8&include_rewards=true&only_new=true

Endpoint to retrieve the top 10 contributions of the last week, based on number of rewards
/api/posts/top?limit=10&start_date=2017-12-1&end_date=2017-12-8&include_rewards=true&sort_by=rewards

Query parameters

• limit How many top projects to receive from the API
• start_date Analyze top projects starting from this date.
• end_date Analyze top projects ending to this day, used in combination with start_date to form a query for retrieving top projects in a given week.
• include_rewards When set to true the Steem API will be queried to aggregate the rewards across the top projects.
• sort_by Can be set to contributions or rewards. When set to rewards then include_rewards must be set to true for this to work. The Steem API will be queried for many more posts to get an accurate projection of top projects based only on rewards.

Technical details

Querying the local database is easy (as seen in the first endpoint sample). The server can aggregate the data quickly in less than a second. When rewards are involved, things are much trickier. We must now query the DB on many posts. The DB backend is MongoDB and data is queried through the aggregation pipeline.

First I tried response streaming but this wasn't playing well at all, it also had the downside of needing to query the blockchain again. This was then replaced with an async model. The API server will return a status and any data it completes.

There is a caching system in place for storing these expensive results. As a result data may become stale after a certain amount of time. So the caching system is capable of expiring the stale data after a certain amount of time, allowing for fresh data to be retrieved. The caching system is smart in regards to the parameters passed in and knows whether or not it is cached based on the query parameters parameter. A weak hash (md5) creates a unique identifier of the cache data in the backend. Caching happens entirely transparently to the user. Checks are added if the data has already been paid out or not, to avoid querying the Steem API if necessary.

A rate limiting system is in place as well to prevent abuse of this endpoint, but only applies if data needs to be queried from the blockchain. The limit is tied to a Steem account to prevent circumventing the system. The date range is limited to 8 days maximum and a maximum limit of 100 projects.

Refactoring

A universal post updater is now in use, allowing data queried from the Steem API to update the stale data in the DB, greatly simplifying the process and reducing coupling of the previous review function.

Bug fixes

• No longer does the front end lock you out when you revoke GitHub access to Utopian
• The auth system now uses refresh tokens, allowing the backend to nifty new things
• Querying the DB for flagged posts no longer returns an empty set

Commits

All commits can be found at https://github.com/utopian-io/api.utopian.io/commits/master

• https://github.com/utopian-io/api.utopian.io/commit/ecb9226d268394f7599dcc9f2be816ef1d74d74d
• https://github.com/utopian-io/api.utopian.io/commit/957c676c44110771b954f3a2638d9d19864b89bc
• https://github.com/utopian-io/api.utopian.io/commit/3226516efdbd1cd1922c0abce564d3efbdba84e4
• https://github.com/utopian-io/api.utopian.io/commit/4e87df768648aabdab855e7b0d78bc65bd7a97d7
• https://github.com/utopian-io/api.utopian.io/commit/4e87df768648aabdab855e7b0d78bc65bd7a97d7
• https://github.com/utopian-io/api.utopian.io/commit/0d5579ac637ef47490e093c7e0d181e6fa7f56dd
• https://github.com/utopian-io/api.utopian.io/commit/acbcb7faa9fca5053d3572f4cf14129cef31cbed
• https://github.com/utopian-io/api.utopian.io/commit/8c32f94802f7145c544842ca64f4cab8eb37e553
• https://github.com/utopian-io/api.utopian.io/commit/c600df2930fe25d399722ca7109d62a6467c5ab1
• https://github.com/utopian-io/api.utopian.io/commit/c215b6518f5132540053e0ea3284b1ffcd316c9e
• https://github.com/utopian-io/api.utopian.io/commit/92678f38f05d85525dcccc3ee95163f724372a31
• https://github.com/utopian-io/api.utopian.io/commit/8abf57b4ed86ea7c4144ad90ec5f56c64193f7b0
• https://github.com/utopian-io/api.utopian.io/commit/b879ec9e4dc637bb1592611cc697fb99f88b9115