What a day it was resolving issues with Hive Engine nodes

I did not expect my weekend to go like this. I actually had some plans for this weekend and everything got changed because of an issue with Hive Engine nodes today. When I woke up in the morning, I saw a lot of messages and tags in the Hive Engine witness chat channel. It is usually practiced for me to take a look at this channel every single day. Today also I did the same and was surprised to see that the nodes had been down for several hours already.

The main origin of the issue was because one of the Hive Node operators was testing out an HF26's feature for some time to see if something was breaking. Unfortunately, it broke Hive Engine nodes and most of the nodes had to restore the database from a backup. It is really a painful process to restore the database from backups. Mongo should improve a lot in that aspect.

Usually, when there is a problem, only a few nodes will get affected and it would be easy for me to restore the other nodes from one of the existing working nodes. But to my surprise, all the witness nodes I manage had issues and there was only one node that did not have any problem luckily and that was a light node by the way. But I was happy with that because I was able to at least restore some of the nodes with the help of a DB backup from that node.

image.png

Usually, when something like this happens, the withdrawal system will also get affected. There will be delays in verifications and withdrawals to get processed. That is the right thing because we will not know if the transactions are correct or not if the verification part is not happening. Verification is nothing by validating the data against multiple nodes. The issue is fully not over. There are still over 9000 blocks to get verified at the time of writing this article. But this is far better than the 20k blocks we had a few hours back.

It is really sad to see this happening on a weekend or when people are on their vacation. It becomes a challenge to fix the issue as soon as possible. Some people already quit running witness nodes saying that the revenue from the nodes is not very great and they think it is not worth running the node. Some people question decentralization and they don't want to run nodes. I and a few members have been here from day 1 and I really enjoy being on this channel and being one of the fellow witnesses.

I have some dApps that are dependent on Hive Engine nodes and it is really important for me to keep the nodes intact. I also offer this as a service to some more people. People who would like to run a node approach me and I offer them as a service. I take care of managing them if there are issues or situations like this. It becomes a headache when there is a need to restore the database again. This is the saddest part here.

Overall it was a fun day and I had to do multiple restores and multiple checks to bring some witnesses up and running. I'm still not done. I have some full nodes to be restored from scratch yet.


If you like what I'm doing on Hive, you can vote me as a witness with the links below.

Vote @balaz as a Hive Witness

Vote @kanibot as a Hive Engine Witness



Posted with STEMGeeks



0
0
0.000
9 comments
avatar

Nice one that you managed to restore and get things working again! The Hive community appreciates your work!

0
0
0.000
avatar

Yikes appreciate your work you do without people like you hive wouldn't be all that much. I'm sure you don't have time for many other projects do you?

0
0
0.000
avatar

Congratulations @bala41288! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s):

You received more than 26000 HP as payout for your posts, comments and curation.
Your next payout target is 27000 HP.
The unit is Hive Power equivalent because post and comment rewards can be split into HP and HBD

You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word STOP

0
0
0.000
avatar

This post has been manually curated by @steemflow from Indiaunited community. Join us on our Discord Server.

Do you know that you can earn a passive income by delegating to @indiaunited. We share 100 % of the curation rewards with the delegators.

Here are some handy links for delegations: 100HP, 250HP, 500HP, 1000HP.

Read our latest announcement post to get more information.

image.png

Please contribute to the community by upvoting this comment and posts made by @indiaunited.

0
0
0.000
avatar

Db backup it needs to be done always!
!1UP

You can earn passive income by delegation of tribe tokens to "The Cartel".

dlmmqb-TheCartel-banner
Click this banner to join "The Cartel" discord server to know more.

0
0
0.000
avatar
(Edited)

I guess this is among the most stressful part of your job. Been able to fix 11k out of 20k is a great one. I hope it is all resolved now.Well done to you and your team. Will vote you as a hive witness

0
0
0.000