Merge #13033: Build txindex in parallel with validation

9b2704777c [doc] Include txindex changes in the release notes. (Jim Posen)
ed77dd6b30 [test] Simple unit test for TxIndex. (Jim Posen)
6d772a3d44 [rpc] Public interfaces to GetTransaction block until synced. (Jim Posen)
a03f804f2a [index] Move disk IO logic from GetTransaction to TxIndex::FindTx. (Jim Posen)
e0a3b80033 [validation] Replace tx index code in validation code with TxIndex. (Jim Posen)
8181db88f6 [init] Initialize and start TxIndex in init code. (Jim Posen)
f90c3a62f5 [index] TxIndex method to wait until caught up. (Jim Posen)
70d510d93c [index] Allow TxIndex sync thread to be interrupted. (Jim Posen)
94b4f8bbb9 [index] TxIndex initial sync thread. (Jim Posen)
34d68bf3a3 [index] Create new TxIndex class. (Jim Posen)
c88bcec93f [db] Migration for txindex data to new, separate database. (Jim Posen)
0cb8303241 [db] Create separate database for txindex. (Jim Posen)

Pull request description:

  I'm re-opening #11857 as a new pull request because the last one stopped loading for people

  -------------------------------

  This refactors the tx index code to be in it's own class and get built concurrently with validation code. The main benefit is decoupling and moving the txindex into a separate DB. The primary motivation is to lay the groundwork for other indexers that might be desired (such as the [compact filters](https://github.com/bitcoin/bips/pull/636)). The basic idea is that the TxIndex spins up its own thread, which first syncs the txindex to the current block index, then once in sync the BlockConnected ValidationInterface hook writes new blocks.

  ### DB changes

  At the suggestion of some other developers, the txindex has been split out into a separate database. A data migration runs at startup on any nodes with a legacy txindex. Currently the migration blocks node initialization until complete.

  ### Open questions

  - Should the migration of txindex data from the old DB to the new DB block in init or should it happen in a background thread? The downside to backgrounding it is that `getrawtransaction` would return an error message saying the txindex is syncing while the migration is running.

  ### Impact

  In a sample size n=1 test where I synced nodes from scratch, the average time [Index writing](https://github.com/bitcoin/bitcoin/blob/master/src/validation.cpp#L1903) was 3.36ms in master and 1.72ms in this branch. The average time between `UpdateTip` log lines for sequential blocks between 400,000 and IBD end on mainnet was 0.297204s in master and 0.286134s in this branch. Most likely this is just variance in IBD times, but I can try with some more trials if people want.

Tree-SHA512: 451fd7d95df89dfafceaa723cdf0f7b137615b531cf5c5035cfb54e9ccc2026cec5ac85edbcf71b7f4e2f102e36e9202b8b3a667e1504a9e1a9976ab1f0079c4

1 files changed, 171 insertions, 0 deletions

diff --git a/src/txdb.cpp b/src/txdb.cpp
index 45ce94ae4..333d3596c 100644
--- a/src/txdb.cpp
+++ b/src/txdb.cpp
@@ -22,6 +22,7 @@ static const char DB_COIN = 'C';
 static const char DB_COINS = 'c';
 static const char DB_BLOCK_FILES = 'f';
 static const char DB_TXINDEX = 't';
+static const char DB_TXINDEX_BLOCK = 'T';
 static const char DB_BLOCK_INDEX = 'b';
 
 static const char DB_BEST_BLOCK = 'B';
@@ -424,3 +425,173 @@ bool CCoinsViewDB::Upgrade() {
     LogPrintf("[%s].\n", ShutdownRequested() ? "CANCELLED" : "DONE");
     return !ShutdownRequested();
 }
+
+TxIndexDB::TxIndexDB(size_t n_cache_size, bool f_memory, bool f_wipe) :
+    CDBWrapper(GetDataDir() / "indexes" / "txindex", n_cache_size, f_memory, f_wipe)
+{}
+
+bool TxIndexDB::ReadTxPos(const uint256 &txid, CDiskTxPos& pos) const
+{
+    return Read(std::make_pair(DB_TXINDEX, txid), pos);
+}
+
+bool TxIndexDB::WriteTxs(const std::vector<std::pair<uint256, CDiskTxPos>>& v_pos)
+{
+    CDBBatch batch(*this);
+    for (const auto& tuple : v_pos) {
+        batch.Write(std::make_pair(DB_TXINDEX, tuple.first), tuple.second);
+    }
+    return WriteBatch(batch);
+}
+
+bool TxIndexDB::ReadBestBlock(CBlockLocator& locator) const
+{
+    bool success = Read(DB_BEST_BLOCK, locator);
+    if (!success) {
+        locator.SetNull();
+    }
+    return success;
+}
+
+bool TxIndexDB::WriteBestBlock(const CBlockLocator& locator)
+{
+    return Write(DB_BEST_BLOCK, locator);
+}
+
+/*
+ * Safely persist a transfer of data from the old txindex database to the new one, and compact the
+ * range of keys updated. This is used internally by MigrateData.
+ */
+static void WriteTxIndexMigrationBatches(TxIndexDB& newdb, CBlockTreeDB& olddb,
+                                         CDBBatch& batch_newdb, CDBBatch& batch_olddb,
+                                         const std::pair<unsigned char, uint256>& begin_key,
+                                         const std::pair<unsigned char, uint256>& end_key)
+{
+    // Sync new DB changes to disk before deleting from old DB.
+    newdb.WriteBatch(batch_newdb, /*fSync=*/ true);
+    olddb.WriteBatch(batch_olddb);
+    olddb.CompactRange(begin_key, end_key);
+
+    batch_newdb.Clear();
+    batch_olddb.Clear();
+}
+
+bool TxIndexDB::MigrateData(CBlockTreeDB& block_tree_db, const CBlockLocator& best_locator)
+{
+    // The prior implementation of txindex was always in sync with block index
+    // and presence was indicated with a boolean DB flag. If the flag is set,
+    // this means the txindex from a previous version is valid and in sync with
+    // the chain tip. The first step of the migration is to unset the flag and
+    // write the chain hash to a separate key, DB_TXINDEX_BLOCK. After that, the
+    // index entries are copied over in batches to the new database. Finally,
+    // DB_TXINDEX_BLOCK is erased from the old database and the block hash is
+    // written to the new database.
+    //
+    // Unsetting the boolean flag ensures that if the node is downgraded to a
+    // previous version, it will not see a corrupted, partially migrated index
+    // -- it will see that the txindex is disabled. When the node is upgraded
+    // again, the migration will pick up where it left off and sync to the block
+    // with hash DB_TXINDEX_BLOCK.
+    bool f_legacy_flag = false;
+    block_tree_db.ReadFlag("txindex", f_legacy_flag);
+    if (f_legacy_flag) {
+        if (!block_tree_db.Write(DB_TXINDEX_BLOCK, best_locator)) {
+            return error("%s: cannot write block indicator", __func__);
+        }
+        if (!block_tree_db.WriteFlag("txindex", false)) {
+            return error("%s: cannot write block index db flag", __func__);
+        }
+    }
+
+    CBlockLocator locator;
+    if (!block_tree_db.Read(DB_TXINDEX_BLOCK, locator)) {
+        return true;
+    }
+
+    int64_t count = 0;
+    LogPrintf("Upgrading txindex database... [0%%]\n");
+    uiInterface.ShowProgress(_("Upgrading txindex database"), 0, true);
+    int report_done = 0;
+    const size_t batch_size = 1 << 24; // 16 MiB
+
+    CDBBatch batch_newdb(*this);
+    CDBBatch batch_olddb(block_tree_db);
+
+    std::pair<unsigned char, uint256> key;
+    std::pair<unsigned char, uint256> begin_key{DB_TXINDEX, uint256()};
+    std::pair<unsigned char, uint256> prev_key = begin_key;
+
+    bool interrupted = false;
+    std::unique_ptr<CDBIterator> cursor(block_tree_db.NewIterator());
+    for (cursor->Seek(begin_key); cursor->Valid(); cursor->Next()) {
+        boost::this_thread::interruption_point();
+        if (ShutdownRequested()) {
+            interrupted = true;
+            break;
+        }
+
+        if (!cursor->GetKey(key)) {
+            return error("%s: cannot get key from valid cursor", __func__);
+        }
+        if (key.first != DB_TXINDEX) {
+            break;
+        }
+
+        // Log progress every 10%.
+        if (++count % 256 == 0) {
+            // Since txids are uniformly random and traversed in increasing order, the high 16 bits
+            // of the hash can be used to estimate the current progress.
+            const uint256& txid = key.second;
+            uint32_t high_nibble =
+                (static_cast<uint32_t>(*(txid.begin() + 0)) << 8) +
+                (static_cast<uint32_t>(*(txid.begin() + 1)) << 0);
+            int percentage_done = (int)(high_nibble * 100.0 / 65536.0 + 0.5);
+
+            uiInterface.ShowProgress(_("Upgrading txindex database"), percentage_done, true);
+            if (report_done < percentage_done/10) {
+                LogPrintf("Upgrading txindex database... [%d%%]\n", percentage_done);
+                report_done = percentage_done/10;
+            }
+        }
+
+        CDiskTxPos value;
+        if (!cursor->GetValue(value)) {
+            return error("%s: cannot parse txindex record", __func__);
+        }
+        batch_newdb.Write(key, value);
+        batch_olddb.Erase(key);
+
+        if (batch_newdb.SizeEstimate() > batch_size || batch_olddb.SizeEstimate() > batch_size) {
+            // NOTE: it's OK to delete the key pointed at by the current DB cursor while iterating
+            // because LevelDB iterators are guaranteed to provide a consistent view of the
+            // underlying data, like a lightweight snapshot.
+            WriteTxIndexMigrationBatches(*this, block_tree_db,
+                                         batch_newdb, batch_olddb,
+                                         prev_key, key);
+            prev_key = key;
+        }
+    }
+
+    // If these final DB batches complete the migration, write the best block
+    // hash marker to the new database and delete from the old one. This signals
+    // that the former is fully caught up to that point in the blockchain and
+    // that all txindex entries have been removed from the latter.
+    if (!interrupted) {
+        batch_olddb.Erase(DB_TXINDEX_BLOCK);
+        batch_newdb.Write(DB_BEST_BLOCK, locator);
+    }
+
+    WriteTxIndexMigrationBatches(*this, block_tree_db,
+                                 batch_newdb, batch_olddb,
+                                 begin_key, key);
+
+    if (interrupted) {
+        LogPrintf("[CANCELLED].\n");
+        return false;
+    }
+
+    uiInterface.ShowProgress("", 100, false);
+
+    LogPrintf("[DONE].\n");
+    return true;
+}