What is MIRA?
I’m @vandeberg, the Senior Blockchain Engineer at Steemit and today I want to talk a little bit about MIRA, the software solution to Steem’s hardware scalability challenge that I have been developing for the last few of months and which we’ve just announced has begun its soft roll-out on Steemit’s nodes. You can find that announcement on @steemitblog. You can also watch me, or listen to me, talk about MIRA on the latest episode of the Steemit podcast below.
Thanks to MIRA we are seeing dramatic reductions in the cost of running Steem nodes that results from migrating the Steem blockchain database (steemd) from expensive, high-performance, hardware to low cost, run-of-the-mill hardware like network attached SSDs or even old school spinning disk drives!
Database Replacement
MIRA is a complete replacement for the backend database that utilizes technology called RocksDB. RocksDB was developed by Facebook to power their Feed which has to load data very rapidly in order to provide a pleasant user experience. Leveraging RocksDB allows us to run the Steem blockchain much more cost effectively, and put our hardware scaling challenges at rest once and for all.
Commodity Hardware
Initially when Steem launched, the database was stored on more affordable hardware. But as the database grew in size, those storage media had difficulty keeping pace with the level of engagement that was being demanded of the protocol. To address those issues we developed innovative software solutions that migrated the database to more high performance storage media in order to ensure a consistent user experience.
This was a good temporary solution, but as Steem continued to grow, the cost of running the blockchain in state-of-the-art storage media started to become financially burdensome on those who run Steem nodes like app developers and Witnesses (i.e. block producers). MIRA resets the hardware requirements for Steem to what they were when Steem first launched, without negatively affecting performance.
Costs Decreasing Over Time
Thanks to MIRA, the technology is now in place to allow Steem to scale much more efficiently into the future, ideally on a course that will see hardware outpacing the requirements of Steem. That means that despite the fact that Steem will continue to grow, it might actually get cheaper to run Steem over time, as long as hardware improvements continue at their current rate. This is important because the largest cost for blockchains (especially blockchains that produce blocks very rapidly) is going to be disk space, but enabling the data to be stored on a slow disk should neutralize that cost.
What is RocksDB?
RocksDB is a fork of LevelDB which is a fork of Berkeley DB. Many blockchains use LevelDB to run on disk, but that’s because they aren’t nearly as fast as Steem, which has 3 second block times. RocksDB adds many more layers of caching algorithms that make it a lot more efficient than LevelDB while also providing interfaces that made it much easier for us to into it into the existing Steem codebase. All of that is important for ensuring that such a high performance blockchain as Steem can be run on cheap hardware and not just exotic hardware like nVMEs.
But this is not a problem that will be unique Steem. As other blockchains like Ethereum seek to shrink their block times, the requirements for quick data access will become all the more important, at which point they will need to look at better scaling solutions for their backend database. Luckily for those team, since MIRA will be Open Source, as much as 90% of their work will already be done for them.
Why an Adapter?
There are projects like Hyperledger use RocksDB directly, which begs the question of why we did so much work to build yet another piece of software. One of the things the adapter (i.e. MIRA) accomplishes is it reduces programmer error, we have interfaces that are well defined and work and are really really easy to develop on, a big part of Steem’s initial 3 month development time can be attributed to those interfaces, they have really good C++ bindings that obfuscate all the database management and allow us to develop all the code very quickly. If we were to build directly on RocksDB, all of the Steem code would have to be rewritten, it would have been very error prone and would have taken months–to years–to do a refactor of that scale and test and ensure no bugs
Non-Steem Use
One of the great things about MIRA is that it will be fully Open Source, so if another team wanted to use MIRA, they would only need to do a little work to integrate MIRA into their solution because MIRA is also blockchain agnostic. It can be used for many different applications. I know that if I were developing an app in C++ and I needed the type of the type of database scaling solution that MIRA provides I would not hesitate to use it and doing so would save me probably 90% of my development time in part because we put a lot of effort into building good interfaces that make it easy to work with MIRA.
The point of MIRA was to wrap RocksDB in an interface that matches what is provided by boost multi-index containers which are a more widely used library, as Boost is standard in C++ applications. That’s because the people who maintain Boost are some of the most talented C++ developers in the world. Without their amazing work, MIRA wouldn’t even be possible. Most of our work was digging into the boost multi-index containers, looking at their implementation and swapping out code where needed to interface with RocksDB rather than be in memory object structures that boost multi index containers do. So if any project want to use Boost to migrate a database from memory to disk, MIRA can help.
The Joy of Open Source Development
The joy of working in Open Source is that we get to borrow great ideas from great engineers, add on our own ideas, and let the rest of the world use all of it. The code is just sitting there, waiting for people to use it. It would be awesome if other projects use MIRA, but even if they don’t, it fills a critical need within our ecosystem. If the Steem source code is the only code that uses MIRA, that will mean that everyone who runs a Steem node will be using MIRA to make their lives easier and reduce their costs. And that’s still a pretty big deal.
$rewarding 100% 15 min
Posted using Partiko Android
Reading about MIRA, this will definitely decrease cost to run blockchain and helps more people to help to be part of steem development. We are grateful for all the developers like you.
Posted using Partiko Android
Wake me up when we have communities :)
To listen to the audio version of this article click on the play image.
Brought to you by @tts. If you find it useful please consider upvoting this reply.
It will be much faster loading for endless scroll if there are no ads. But yes I know ads are now relevant for profitability so we do understand that. Thank you for this update and will check out MIRA as another promising SMT to watch out for.
Couldn't the government narrow down who's running a node in DPoS? I was reading how centralized nodes aren't very decentralized.
dPOS is centralised to a permissioned set of block producers, right. When they speak about "decentralisation" they mean that they have lowered the economic hurdles for you to take part in this political process of becoming a blockproducer. So technically it does not decentralize anything. But dPOS is obviously not relying on topological decentralization, it relies on delegating power to representatives of which 21 will produce the blocks.
Since voting in Steem is plutocratic (dependent on your money) it is not democratic and heavily biased towards creating circles of power and influence. The "government" does not wnat to discuss this aspect...obviously they are not interested in "power to the users".
Goi g to check this out now 😀
Posted using Partiko iOS
So you're telling me EOS will be screwed by scaling (like Steem did, until RocksDB/Mira go implemented)? 😃
Well "screwed" is a difficult term. One could argue that as long as the EOS stakeholders are making money, they're not screwed. Let's put it this way; the EOS main chain is already over 5TB and it's still all in RAM. A big part of the reason why it's so big is because it produces blocks every half a second. But none of this is necessarily bad for Block.One because they don't run the chain or run an app on the chain. The people who will get screwed are the suckers who choose to build on it ;)
5TB !!! That's insane. I thought EOS was going to produce every 3 seconds like Steem.
And growing! Rapidly! Good thing they spammed all those transactions ;). They need to produce blocks that frequently if they want to deliver the kind of performance they are claiming EOS is capable of. Imagine someone wants to string together 20 smart contracts all of which need to be executed sequentially, before funds are transferred. That one transfer is still going to take 10 seconds to occur even with .5 second block times. Had they chosen 3 second block times... well you can do the math. Not saying it's a bad solution to the problem they're choosing to tackle, but it highlights part of the reason we're so committed to application-specificity. For Steem, 3 second blocks work great and few see any reason to increase that rate, especially bearing in mind that doing so would only make the chain grow faster.
That is great work and I hope there will be some updates once thing settle to look at the numbers in regards to performance and savings.
First of all this is a monumental moment in Steem's history, the costs of running 3 year old blockchain will be cheaper than ever, I don't think many people understand this. Every blockchain that gets bigger over time is more costly to maintain, with MIRA Steem can be run by almost any PC configuration.
I have one question though, are there any bottlenecks left that would prevent Steem for massive influx of people, something that would potentially be another 1year development cycle problem?
I've always wondered about the whole 'logistics' of steem/blockchains; of course modern computer processing speeds are hard to even wrap your head around, I can't help but sometimes wonder how 'long term' feasible block chain technology is... I mean every three seconds there's more data to be processed. Admittedly I'm not a computer scientist and I'll leave this speculation up to people that are more informed on the subject. But I like what you're saying!
Thanks for your nice summary and very honest interview. Good to see you on Steemit as well as GH :)
ps. don't forget to merge the 14-week powerdown fix haha :)
I really dont care to educate myself on what is what. All i care about is one word:
and how @andrarchy can take that word and mold it into a 100 different versions, comparisons with other blockchains to show twitter, facebook, Reddit how STEEM compares.
Doesnt matter how it works. All that matters is that it does work and that the shill info is so good that the receiving end doesnt even mind, because its just so beautiful..
One more reason to continue working with steem blockchain.
Posted using Partiko iOS
Congratulations @vandeberg! You have completed the following achievement on the Steem blockchain and have been rewarded with new badge(s) :
You can view your badges on your Steem Board and compare to others on the Steem Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPDo not miss the last post from @steemitboard:
Vote for @Steemitboard as a witness to get one more award and increased upvotes!
@vandeberg, you are a rock star. Thks for your contribution to steem blockchain and to the world. I remember chatting with you during steemfest 2. Not many people get to know the engineer behind such project. Congratulation to the team for MIRA roll-out.
Well played! This sounds like a huge milestone accomplished.
Great work!
This story was recommended by Steeve to its users and upvoted by one or more of them.
Check @steeveapp to learn more about Steeve, an AI-powered Steem interface.
Thanks for explaining more about the recent adoption of the steem blockchain. I hope it will more positive value to our community
This is a great breakdown for even those of us that are less technical. I would love to eventually run a node or even be a witness to support the further decentralization of the protocol and this makes it easier!
Posted using Partiko iOS
Congratulations to all involved in the development of MIRA.
This is indeed a red letter day and an amazing transformation in the effectiveness of Steemit Inc in delivering what the Steem community really needed from them.
I posted about how important the ability to run a Steem node on commodity hardware was immediately after Steemit Inc announced the 70% cuts.
You have really turned things around. Well done.
Where is the github code for MIRA? Thank you
Edit: I think I found it:
https://github.com/steemit/steem/tree/8dc43704f7abf65b5ca9e737b965de964be9779c/libraries/mira
You're very eloquent @vandeberg! Congrats on this remarkable work and I hope we'll get to hear you more in the future!
The best part of this comment is that Steemit used to have a real-life Dale Gribble, too.
This post has been included in the latest edition of The Steem News - a compilation of the key news stories on the Steem blockchain.
Oh the irony of this development. I'm glad that the scaling issue is being solved. The fact that a database technology that was developed by Facebook (arch-enemy of Steem.... Hahaha) and is now going to be what possibly saves the day for Steemit scaling. Congrats gentlemen.
Bravo Michael!
To me, this is BY FAR the most important technical Steem improvement, ever. Exceptional work, keep it up!
Kind regards, @scipio & @steem-ua
FYI, I included this post in my Interesting Links post for May 8. You'll receive beneficiary rewards for 5% of that post's payout.
Very nice @vandeberg, to find this post and listen to your explanation of what MIRA represents. While I do not have the level of expertise you cite, I have worked with normalized data structures for years. Through the need to understand what an application like MS Access can provide in the way of business solutions (like currently working on one to handle our family’s tax reporting requirements …).
So … This was more than a “casual” post reading for me.
Given your background and role, I have a question which is very important to me as an investor. It pertains to the whole “open source” approach to solving “real world” problems in advancing the capabilities of this new “digital asset” class. My question is how much confidence you have in yourself and your fellow “open source” co-workers that no “trojan horse” code could be introduced into it somehow / some way by a “hostile” agent of some sort?
My concern is that this is theoretically possible. I won’t take your valuable time to “spell out” what I mean by hostile, as I imagine (I hope …) you may have thought of those possibilities yourself. What I am asking is for your answer to the question, with the hope of receiving some assurance there are some very good programmers who are ever watchful, for just this possibility. Sort of along the lines of the famous quote:
My sincere thanks to you and @andrarchy, for the effort to get this post made and out for us. Supported and resteemed, to do my small part to improve the understanding, of those of our fellow Steemians who may be interested, of how our Steem blockchain works "under the hood" ...
P.S. Also bookmarked in SteemPeak … 😉 … for future reference ...
good job !!!!!!
Gsksgeitecajcs
Posted using Partiko Android
Can you open with:
MIRA: Steemit on the Rocks
or something? :)
That's nice! I rented a server with 160GB SSD, but the witness installer that I found load a very big block_log, that doesen't fit on the SSD. Now Iam installing the github steemd, which last 4 days now and is still at half of the chain. Where do I find that mira update or alternatives for my server?
This is awesome @vandeberg
Posted using Partiko Android
комментарий ручками
post somethin' weekly, ty