Skip to main content

Tutorial: Creating an Indexer

In this tutorial, we will build an indexer using the NEAR Indexer Framework. The indexer will listen realtime blocks data from NEAR blockchain.

To get our indexer up and running we will need two steps:

  1. To initialize the indexer
  2. To start it

The full source code for the indexer example is available in the GitHub repository.

important

Source code link is for nearcore repository, as the Indexer Framework is part of the nearcore codebase. We provide the link to master branch. If you want to use the latest stable release version you should check the releases page and checkout the corresponding tag.

danger

NEAR Indexer Framework only works on Linux x86, it does not support Windows or MacOS

Prerequisites​

Install Rust​

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Install developer tools​

apt update
apt install -y git binutils-dev libcurl4-openssl-dev zlib1g-dev libdw-dev libiberty-dev cmake gcc g++ python docker.io protobuf-compiler libssl-dev pkg-config clang llvm cargo awscli
danger

NEAR Indexer Framework only works on Linux x86, it does not support Windows or MacOS

Clone nearcore repository​

git clone git@github.com:near/nearcore.git

Initialization​

In order for our indexer to process blocks it needs to join the NEAR network as a node. To do that, we need first to initialize it, which will download the blockchain genesis config, and create a key for our node to communicate with other nodes.

Go to the nearcore/tools/indexer/example folder and build the indexer:

cd nearcore/tools/indexer/example && cargo build --release

Then, run the following command to initialize the network configuration:

  cargo run --release -- --home-dir ~/.near/localnet init

Depending on the network we want to connect, the keys will be created in different folders (~/.near/<network>).

Config File​

A configuration file (~/.near/<network>/config.json) is created automatically, whoever, it is recommended to replace with one of the following ones, intended for RPC nodes:

Configuration Options

See the Custom Configuration section below to learn more about further configuration options.

Starting the Indexer​

After we finish initializing the indexer, and configuring it, we can start it by running the following command:

  cargo run --release -- --home-dir ~/.near/localnet

Run into an Error?​

  • If your indexer cannot find boot nodes, check the boot nodes section

Parsing the Block Data​

From the block data, we can access the transactions, their receipts and actions. See the code below for an example of how to parse the block data:

Custom Configuration​

By default, nearcore is configured to do as little work as possible while still operating on an up-to-date state. Indexers may have different requirements, so you might need to tweak the configuration based on yours.

Shards/Accounts to Track​

We need to ensure that NEAR Indexer follows all the necessary shards, so by default the "tracked_shards_config" is set to "AllShards". The most common tweak you might need to apply is listing to specific shards; to do that, lists all the shard UIDs you want to track in the "tracked_shards_config" section (~/.near/<network>/config.json file):

...
"tracked_shards_config": {
       "Shards": [
              "s3.v3",
              "s4.v3"
       ]
},
...

Or, if you want to track specific accounts:

...
"tracked_shards_config": {
       "Accounts": [
              "account_a",
              "account_b"
       ]
},
...

Sync Mode​

You can choose Indexer Framework sync mode by setting what to stream:

  • LatestSynced - Real-time syncing, always taking the latest finalized block to stream
  • FromInterruption - Starts syncing from the block NEAR Indexer was interrupted last time
  • BlockHeight(u64) - Specific block height to start syncing from.

Streaming Mode​

You can choose Indexer Framework streaming mode by setting what to stream:

  • StreamWhileSyncing - Stream while node is syncing
  • WaitForFullSync - Don't stream until the node is fully synced

Finality​

You can choose finality level at which blocks are streamed:

  • None - optimistic, a block that (though unlikely) might be skipped
  • DoomSlug - near-final, a block that is irreversible, unless at least one block producer is slashed
  • Final - final, the block is final and irreversible.

Boot Nodes​

If your node can't find any peers to connect to, you can manually specify some boot nodes in the config.json file. You can get a list of active peers for your network by running the following command:

curl -X POST https://rpc.testnet.near.org \
  -H "Content-Type: application/json" \
  -d '{
        "jsonrpc": "2.0",
        "method": "network_info",
        "params": [],
        "id": "dontcare"
      }' | \
jq '.result.active_peers as $list1 | .result.known_producers as $list2 |
$list1[] as $active_peer | $list2[] |
select(.peer_id == $active_peer.id) |
"\(.peer_id)@\($active_peer.addr)"' |\
awk 'NR>2 {print ","} length($0) {print p} {p=$0}' ORS="" | sed 's/"//g'

And then add the output to the boot_nodes section of your config.json file as a string:

...
"network": {
    "addr": "0.0.0.0:24567",
    "boot_nodes": "ed25519:8oVENgBp6zJfnwXFe...",
    ...
},
...

Historical Data​

Indexer Framework also exposes access to the internal APIs (see Indexer::client_actors method), so you can fetch data about any block, transaction, etc, yet by default, nearcore is configured to remove old data (garbage collection), so querying the data that was observed a few epochs before may return an error saying that the data is not found. If you only need blocks streaming, you don't need this tweak, but if you need access to the historical data right from your Indexer, consider updating "archive" setting in config.json to true:

...
"archive": true,
...

Using NEAR Indexer in your Project​

You can also use NEAR Indexer Framework as a dependency in your own Rust project. To do that, add the following to your Cargo.toml file (replace 2.8.0 with the latest stable release version):

[dependencies]
near-indexer = { git = "https://github.com/near/nearcore", tag = "2.8.0" }
near-indexer-primitives = { git = "https://github.com/near/nearcore", tag = "2.8.0" }
near-config-utils = { git = "https://github.com/near/nearcore", tag = "2.8.0" }
near-o11y = { git = "https://github.com/near/nearcore", tag = "2.8.0" }
near-primitives = { git = "https://github.com/near/nearcore", tag = "2.8.0" }

Looking for Support?

If you have any questions, connect with us on Telegram or Discord .

Happy coding! 🚀