Deploying an indexer to production

This page will show you how to deploy an indexer. To follow this tutorial, you will need a server with Docker installed.

We will deploy the following components:

• Etcd: a distributed Key-Value store, it is used by Apibara to store the indexer's progress and to ensure only one instance of each indexer is running at the same time.
• Indexer: the indexer you developed.

Note: for a production-grade deployment, you will need to deploy 3 or 5 Etcd instances on different machines to ensure liveness of the cluster. If you don't want to manage the cluster yourself, you can use the Apibara hosted service to deploy your indexers following all the best practices.

The Etcd cluster#

We start a single-node Etcd cluster. We use the Bitnami etcd docker image and disable authentication. Data is persisted to the etcd_data volume.

version: '3'

services:
  etcd:
    image: docker.io/bitnami/etcd:3.5
    environment:
      ALLOW_NONE_AUTHENTICATION: yes
    volumes:
    - etcd_data:/bitnami/etcd
    ports:
    - 2379:2379

volumes:
  etcd_data:

The Apibara indexer#

Running the indexer simply requires to start the container and point it to the right script file. We set three environment variables:

• AUTH_TOKEN: the token used to authenticate with the DNA stream.
• PERSIST_TO_ETCD: tell the indexer to persist its state to Etcd. Must contain the URL of the Etcd cluster previously created.
• SINK_ID: the unique indexer id. This is used to resume indexing from where the indexer previously left off and to ensure only one copy of the indexer is running at any given time.

Let's add the following snippet to the existing docker-compose.yml file, just below the etcd service.

indexer:
    image: quay.io/apibara/cli:0.1
    depends_on:
    - etcd
    restart: always
    command:
    - run
    - /data/script.js
    volumes:
    - ${PWD}/script.js:/data/script.js
    environment:
      PERSIST_TO_ETCD: http://etcd:2379
      SINK_ID: my-indexer
      AUTH_TOKEN: ${AUTH_TOKEN}

Notice that in this example the indexer source code is contained in a single file named script.js. If your indexer's code is split across multiple files, you can mount the source directory in the container.

Note: This example uses the apibara/cli image which contains the apibara CLI tool together with all the built-in sinks. For this reason this image is quite large (almost 200MB), you can also use a sink-specific Docker image (named apibara/sink-*) to reduce your container start time.

Running the indexer#

Create a .env file with your DNA token in it:

AUTH_TOKEN=dna_AbC

You can now start the indexer with docker compose up --env-file=.env.

Notice that if you restart the indexer, it may take a while (up to one minute) before it starts because it needs to acquire the Etcd lock. This is done on purpose: running two instances of the same indexer will result in duplicate data inserted in your database or sent to webhooks.