Drizzle's plugin internals

This section describes how the Drizzle plugin works. Understanding the content of this page is not needed for using the plugin.

The plugin wraps all database operations in the transform and factory functions in a database transaction. This ensures that the indexer's state is always consistent and that data is never lost due to crashes or network failures.

More specifically, the plugin is implemented as a middleware.

At a very high level, the plugin looks like the following:

indexer.hooks.hook("handler:middleware", async ({ use }) => {
  use(async (context, next) => {
    await db.transaction(async (txn) => {
      // Assign the transaction to the context, to be accessed using useDrizzleStorage
      context.db = txn;

      await next();

      delete context.db;

      // Update the indexer's state with cursor.
      await updateState(txn);
    });
  });
});

The indexer needs to be able to rollback state after a chain reorganization. The behavior described in this section is only relevant for un-finalized blocks. Finalized blocks don't need special handling since they are, by definition, not going to be part of a chain reorganization.

The main idea is to create an "audit table" with all changes to the indexer's schema.

The name of the audit table is airfoil.reorg_rollback and has the following schema.

+------------+--------------+-----------------------+
| Column     | Type         | Modifiers             |
|------------+--------------+-----------------------|
| n          | integer      |  not null default ... |
| op         | character(1) |  not null             |
| table_name | text         |  not null             |
| cursor     | integer      |  not null             |
| row_id     | text         |                       |
| row_value  | jsonb        |                       |
| indexer_id | text         |  not null             |
+------------+--------------+-----------------------+

The data stored in the row_value column is specific to each operation (INSERT, DELETE, UPDATE) contains the data needed to revert the operation. Notice that the table's row must be JSON-serializable.

At each block, the plugin registers a trigger for each table managed by the indexer. At the end of the transaction, the trigger inserts data into the audit table.

The audit table is periodically pruned to remove snapshots of data that is now finalized.

Reverting a block

When a chain reorganization is detected, all operations in the audit table where cursor is greater than the new chain's head are reverted in reverse order.

• op = INSERT: the row with id row_id is deleted from the table.
• op = DELETE: the row with id row_id is inserted back into the table, with the value stored in row_value.
• op = UPDATE: the row with id row_id is updated in the table, with the value stored in row_value.

The state of the indexer is persisted in the database, in the airfoil.checkpoints and airfoil.filters tables.

The checkpoints table contains the last indexed block for each indexer.

+------------+--------------+-----------------------+
| Column     | Type         | Modifiers             |
|------------+--------------+-----------------------|
| id         | text         |  primary key          |
| order_key  | integer      |  not null             |
| unique_key | text         |                       |
+------------+--------------+-----------------------+

The filters table is used to manage the dynamic filter of factory indexers. It contains the JSON-serialized filter together with the block range it applies to.

+------------+--------------+-----------------------+
| Column     | Type         | Modifiers             |
|------------+--------------+-----------------------|
| id         | text         |  not null             |
| filter     | text         |  not null             |
| from_block | integer      |  not null             |
| to_block   | integer      |  default null         |
+------------+--------------+-----------------------+