A Leaner Hivesearcher and the State of Search on Hive

Hivesearcher is staying. That is the headline, because for a while we genuinely debated shutting it down. Instead we rebuilt it to run on a small fraction of the infrastructure it used to need, as part of a broader round of cost cutting at Ecency and we are lowering our proposal ask on the Decentralized Hive Fund from 150 to 100 HBD per day. This post explains the thinking, what changed under the hood and where Hivesearcher fits now that Hive has more than one way to search.

Three engines, honestly compared

Search on Hive is in better shape than it has ever been and it is worth being precise about what each engine does, because they are not interchangeable.

Hivemind is the social layer that powers feeds, follows and communities on every frontend and as of the current release it has a working text search again: search_api.find_text on the standard JSON-RPC API. It is keyword search over the full history of posts, with relevance, trending and created sorting and an author filter and you can try it interactively at api.hive.blog under the Legacy JSON-RPC API. Hivemind also handles account and community lookup, as it always has.

HiveSense is the semantic engine from the core developers, shipped alongside the standard API node stack and already part of some full hive api nodes. Your query is matched by meaning, not exact words, across roughly a hundred languages. Ask "how do I get my account back" and it finds account recovery guides that never use your phrasing. It also finds posts similar to a given post and authors whose body of work matches a topic. By its developers' description it indexes around 14 million posts, skipping very short content by design and it takes a few seconds per query because semantic search does real work for every question.

Hivesearcher is our keyword engine on Elasticsearch: every post and every comment since 2016, more than 134 million documents, searchable within seconds of hitting the chain, with results in milliseconds. It supports things the other two do not: exact phrases, exclusions, tag, type and date filters, the query language power users and tools rely on:

"account recovery" -giveaway author:good-karma tag:hive type:post

These three are complementary. Semantic search answers "what is this about". Keyword search answers "find exactly this". Name lookup gets you to the right account or community. An ecosystem is healthier with all of them and now it has all of them.

Why we almost retired it and why we did not

Running a dedicated search cluster has a standing cost, in servers and in the attention it takes to monitor, upgrade and fix. With the core developers now shipping search as part of the commons, we seriously considered retiring Hivesearcher and pointing everything at the alternatives.

Two things stopped us. First, the capabilities above have no substitute: full comment coverage, exact-match operators and instant responses matter to power users, researchers and the tools built on our API. Second, the related-content feature it powers, the "Read next" suggestions under posts, quietly does a lot of work: readers use it to go deeper and it helps search engines and crawlers discover older content that would otherwise sit unlinked. Cutting that off to save a modest sum felt like the wrong trade.

So we took the third option: keep the service, strip the cost.

What changed under the hood

Hivesearcher used to run as a multi-server setup with a replicated index, built for a scale of traffic it no longer needs to plan for. We have now consolidated the entire service, the search index, the API, the indexers that follow the chain and the website, onto a single modest server with an attached storage volume. The index went from two full copies down to one and we rebuilt it with a leaner layout while we were at it.

What you give up with a setup like this is redundancy and we want to be transparent about that trade. If the server fails, search is briefly down instead of failing over invisibly and in the worst case the index is rebuilt from source data, which takes about a day. For a free public service, we think that is the right trade and every capability, every API key and the same response speed carry over unchanged. If you use Hivesearcher today, nothing about this change is visible to you.

This is also not the only cut. We have been trimming administrative overhead across Ecency, subscriptions, services and operational costs that have nothing to do with search. The lower proposal ask reflects that whole effort, not any single service.

The proposal: 150 to 100 HBD per day

With these reductions in place, we are lowering the Ecency proposal on the Decentralized Hive Fund from 150 to 100 HBD per day. Ecency's ask has been shrinking steadily as we simplify and this continues that direction: less overhead, same apps, same services, same pace of development on the things that matter (security and bug fixes).

You can support the proposal at ecency.com/proposals/379. Votes are the difference between funded and unfunded and every bit of Hive Power helps.

Run your own, if you want to

Everything Hivesearcher runs on is open source under the Ecency organization on GitHub and the lean single-server shape we just moved to is exactly the shape a community instance would take:

1. hivesearcher-indexer streams posts and comments from any public Hive RPC node into a PostgreSQL table. No HAF or Hivemind node required.

2. hivesearcher-elastic reads that table and keeps an Elasticsearch 8 index in sync, including the initial backfill of the whole chain.

3. hivesearcher-api is the API server: the query syntax, relevance tuning and API key management.

4. hivesearcher is the optional web UI.

One server with ordinary CPU hardware and a few hundred gigabytes of storage is enough for the full history. The initial backfill takes on the order of days and after that it is standard, well documented Elasticsearch operations. If you want an instance for your project or your community, reach out on Hive or through any Ecency channel and we will help you get set up.

Hivesearcher has answered the ecosystem's search queries for the better part of a decade. It now does so at a fraction of its old cost, alongside two core-team engines that cover what it does not. That is a better place than search on Hive has ever been and we would like to keep it that way.

If Hivesearcher's API is useful to your project, you can request a free key at hivesearcher.com and if Ecency's work is useful to you, the proposal vote is the best way to say so.

https://ecency.com/proposals/379