Steem Engine Tender: A Sidechain Explorer - 0.0.4 - Performance

TL;DR: The site should now be more responsive: https://steem-engine.rocks/

There's been a lot of growth in the Steem Engine sidechain. And I wasn't keeping up with it. I had a slight hack that came back to bite me due to this scalability oversight.

The main feature in this release is the addition of a proper account/symbol index (instead of the hack). Prior to this, account and symbol lookups were done ad-hoc, which was fine when the sidechain was under 400,000 blocks. But now that it's over 1,000,000 blocks, something had to be done.

I actually solved this weeks ago, but needed to rebuild the index in order to get it on steem-engine.rocks. In the meantime, my staging solution died (made useless by a single fan failure, actually) and my ISP throttled me, so I wasn't able to get a new index up. Well, those are my excuses.

Because of these roadblocks, I spent some time tweaking the performance of the transaction processing tasks. Either I could throw more hardware at the public server or make the process more efficient with the existing hardware. Why not both?

One of the design goals for tender is for exchanges to be able to run it independently so they don't have to rely on my public version. If they did this, they'd probably be fine because they'd likely have the hardware required, while I simultaneously applied the real fix.

Other features/fixes added:

• Bug fixes, gem updates
• Fixed some UI elements
• Tweaks to contract interpretation
• Added a contract action baseclass that all actions now inherent from.
• Tweaked replay detection
• Improved checkpoint evaluation
• Added "turbo mode" to transaction processing
• Moved redis keyspace reclamation to after commit

Turbo Mode (back-end transaction importing)

For the initial index, I managed to get processing on my staging solution from 30/tps (transactions per second) to 180/tps by deferring commits and adding a limit.

rake tender:trx_ingest[drop_redis_keys,max_transactions,turbo]  # Ingest Steem Engine transactions from Meeseeker

From the project source, run:

$ bundle exec rake tender:trx_ingest[true,1000,true]

Without these options, processing assumes you want all available transactions processed, just like before. But with these options, the above example asserts we only want to process up to 1,000 transactions in memory, then commit them, then drop the keys from redis so they won't be processed on the next pass. This allows the processing to still be interrupted without creating gaps in the index.

As always, you can launch your own tender instance with docker (recommend: run your own local steemd to improve performance):

$ docker run -it --name tender -p 5000:5000 -p 3000:3000 inertia/tender:latest

All changes in v0.0.4 tag: https://github.com/inertia186/tender/commits/v0.0.4 (compare)