Filtering of operations returned by the get_account_history API call.

avatar
(Edited)

gears_blockops_pink.png

OVERVIEW

A very useful Hive non-consensus change was added by the @blocktrades core coding team last week; namely the filtering of operations returned by the "get_account_history" API call.

For those not intimately acquainted with Hive API calls, the "get_account_history" request allows you to obtain the Hive transactions performed by a single Hive account. These are returned in batches of up to 1000 transactions, in reverse chronological order, starting from a transaction number that you define.

Prior to last week a history request would include all transactions for the account with no possibility to filter by type. So if you only wanted to look at the account's power-up transactions, of which there are, say, 3, you would still need to pull though their 10,000 comments, 100,000 votes and 300,000 splinterlands custom json transactions. This made the "get_account_history" API call particularly data-heavy and time consuming.

Following the change it's now possible to filter the results by transaction type, allowing you to grab just the 3 power-up transactions you were looking for.

I've spent the last few months updating my HeyStack project (more on this in a couple of weeks) for which the above change is hugely useful. As such, when I saw it mentioned in @blocktrades' post (https://hive.blog/hive-139531/@blocktrades/update-on-blocktrades-hive-core-coding) I immediately tried to put it into action.

Easier said than done! But after a fair amount of digging around, a little trial and error, and the breaking-out of the BigInt, I got there in the end. Here's what I learned!

NEW PARAMETERS

Prior to the change there were three main parameters:

  • account (string): the account name, e.g. "miniature-tiger";
  • start (int): the starting transaction number of the batch (which can be set to -1 for the most recent transaction); and
  • limit (int): the number of transactions in the batch (a max of 1000).

So if I have made 125,499 transactions (true at the time of writing) then parameters ["miniature-tiger", -1, 0] would have returned my most recent single transaction and ["miniature-tiger", 999, 999] would have returned my first 1000 transactions (you get 1 more transaction than you ask for because Hive is very inclusive, and you count from zero, because arrays).

After the change there are two new parameters:

  • operation_filter_low (int64)
  • operation_filter_high (int64)

They are defined here:
https://gitlab.syncad.com/hive/hive/-/blob/master/libraries/plugins/apis/account_history_api/include/hive/plugins/account_history_api/account_history_api.hpp

These parameters operate a matching bitwise filter based on the transaction list shown here (once the SMT transactions are stripped out):
https://gitlab.syncad.com/hive/hive/-/blob/master/libraries/protocol/include/hive/protocol/operations.hpp

There are two 64 bit parameters: the first one covers the first 64 transaction types, the second one covers the remainder. There are currently 73 transaction types listed.

HOW DOES IT WORK - IN SIMPLE TERMS!

Take the transaction list, strip out the SMT transactions, and use 2^(n-1) of the transaction number in the list for the low parameter.

For example (using transactions from the start of the list):
1: vote_operation: 1 (i.e. 000001)
2: comment_operation: 2 (i.e. 000010)
3: transfer_operation: 4 (i.e. 000100)
4: transfer_to_vesting_operation: 8 (i.e. 001000)

So if you just want transfer transactions and no others then the low parameter would be 4 (and the high parameter would be 0).

Parameters of ["miniature-tiger", 999, 999, 4, 0] would return all the transfer transactions out of my first 1000 transactions (i.e. an array of, say, 2 transactions, or possibly no transactions at all if I hadn't got around to making transfers early in my Hive career).

You will still need to loop through numerous calls if you want all my transfers (e.g. ["miniature-tiger", 1999, 999, 4, 0] for the next 1000 etc etc) but most users will probably only be interested in the latest transactions rather than the whole account history.

For transaction types listed above 64 use the high parameter, deducting 64 from the transaction number. Again from the transactions list:
65: proposal_pay_operation: 2^(65 - 64 -1) = 1
66: sps_fund_operation: 2^(66 - 64 - 1) = 2
i.e. basically the same as for the low parameter but starting the list over from transaction 65.

So if you just want my proposal_pay transactions (I don't got none before you ask) and no others then the low parameter would be 0 and the high parameter would be 1 (i.e. 2^(65-64-1) = 1) and so forth.

Why this complex approach? Because with the bitwise filter if you want both comments and transfers but no other transactions you can set the low parameter to 6 (2 + 4 = 6 i.e. 000110), based on:
2: comment_operation: 2 (i.e. 000010)
3: transfer_operation: 4 (i.e. 000100)

And if you want comments, transfers, and proposal payments then you set the low parameter to 6 and the high parameter to 1.

Super-useful!!!

BREAKING OUT THE BIGINT()

One issue I found coding in js for HeyStack is that Int cannot handle numbers over 2^53-1. So trying to calculate a low parameter number for transactions above 53 in the list (and below 65) won't fly.

There's undoubtedly a sexier way to do this using hexes and other bitwise-shift witchery but I decided just to break out the BigInt().

The code below works to calculate the low and high parameters. Just add your own desired transaction numbers in "paramNumbers".

let paramNumbers = [3, 4, 10, 23, 51, 55, 56, 62, 63, 65];
let lowParam = paramNumbers.filter(x => x <= 64).reduce((total, amount) => total + (2n ** BigInt(amount-1)), 0n);
let highParam = paramNumbers.filter(x => x > 64).reduce((total, amount) => total + (2n ** BigInt(amount-64-1)), 0n);

That's about it! As I said, a very useful change and hopefully the above explanation is useful for anyone trying to use it!



0
0
0.000
4 comments
avatar
(Edited)

I've noticed that not all full nodes are running the plugin that is required for this feature. But one that is: api.hive.blog

Here's an example of how to invoke this from the command-line, using the condenser_api namespace, filtering on the vote_operation bit:

curl -s --data '{
  "jsonrpc": "2.0",
  "method": "condenser_api.get_account_history",
  "params": ["hiveio", 1000, 1000, 1, null],
  "id":1
}' https://api.hive.blog

And here's the same example, using the account_history_api namespace, again filtering on the vote_operation bit:

curl -s --data '{
  "jsonrpc":"2.0",
  "method": "account_history_api.get_account_history",
  "params": {
    "account": "hiveio",
    "start": 1000,
    "limit": 1000,
    "include_reversible": false,
    "operation_filter_low": 1,
    "operation_filter_high": null
  },
  "id":1
}' https://api.hive.blog
0
0
0.000
avatar

I've noticed that not all full nodes are running the plugin that is required for this feature. But one that is: api.hive.blog

Yes. I think that as the code change was made only very recently it could take a little while to be rolled out by all nodes.

curl -s --data '{
"jsonrpc": "2.0",
"method": "condenser_api.get_account_history",
"params": ["hiveio", 1000, 1000, 1, null],
"id":1
}' https://api.hive.blog

If you wanted both high and low parameters (both set to 1) would it be as follows?
"params": ["hiveio", 1000, 1000, 1, 1, null],

0
0
0.000
avatar

No. The final null would become the high bits:

curl -s --data '{
  "jsonrpc": "2.0",
  "method": "condenser_api.get_account_history",
  "params": ["hiveio", 1000, 1000, 1, 1],
  "id":1
}' https://api.hive.blog
0
0
0.000