upstream changes#4
Open
mikehash wants to merge 1191 commits into
Open
Conversation
…ng to ResyncIndex
#1527) * feat(ethrpc): observe JSON-RPC calls for block sync and emit error requests * refactor(ethrpc): naming of the observation method
clear() does not shrink a Go map's underlying bucket allocation, so a single very large block would leave hits at its high-water mark even after reset. Reinitialize with the maxPendingHits pre-size instead so the runtime can release the oversized allocation.
…borts sync could block forever while queueing block hashes if all workers were stuck retrying block not found, because the coordinator stopped reading worker aborts.
stop EVM sync from silently stalling when a backend's tip feed dies. Added subscription watchdogs (Tron poll-only) plus metrics
a2ec3b3 remapped 1inch's four fee tiers onto low/medium/high (discarding the low tier and folding instant into high), so 1inch no longer surfaces a separate instant tier. The panel description still claimed 1inch was a "provider that return[s] four tiers", which is no longer true and would send an on-call engineer chasing a phantom instant-tier discrepancy. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
#1583) * fix(tron): the receipt for unsolidified block should not be set to nil so all token transfers are parsed and indexed correctly * feat(tron): increase number of connections in the HTTP client to speed up syncing * feat(tron): remove zeromq nil check to reduce number of requests per block to speed up syncing * feat(tron): use HTTP endpoint for getting block has by number to fasten sequential sync path * fix(tron): drain HTTP response body before close to enable connection reuse net/http only returns a connection to the idle pool when the response body is read to EOF and then closed. Request() closed the body without draining: the >=300 early return reads nothing, and json.Decoder can leave trailing bytes unconsumed on larger responses, so connections were torn down instead of pooled — defeating the MaxIdleConns/MaxIdleConnsPerHost tuning this branch added to speed up syncing. Drain the body in the deferred close. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(tron): drop dead isSolidified param from requestBlockHashByNum requestBlockHashByNum is only called from GetBlockHash's fast path, which is guarded by isBlockSolidified() && solidityNodeHTTP \!= nil, so the sole caller always passed true and the /wallet/getblock (full-node) branch was unreachable. Hardcode the solidity-node client and /walletsolidity/getblock path so the function's actual single-purpose role is explicit and no untested branch lingers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(tron): add initial sync tuning --------- Co-authored-by: pragmaxim <pragmaxim@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
PIVX Sapling transactions (version >= 3) include shielded fields (valueBalance, vShieldedSpend, vShieldedOutput, bindingSig) after the transparent part. wire.MsgTx.TxHash() only hashes the transparent fields, producing a phantom txid that is stored in the address index. Fix by computing the txid as sha256d of the full raw transaction bytes for Sapling transactions, matching PIVX's consensus definition. Also fix a pre-existing encoding bug in ParseTx: it used WitnessEncoding via Deserialize(), which can misinterpret a Sapling tx with 0 inputs as a segwit transaction (0x00 0x01 marker). Now reads the version first and uses BaseEncoding for version >= 3, matching PivxDecodeTransactions. Fixes #1522
…ndex EthereumRPC.GetTransaction relied solely on eth_getTransactionByHash, which returns null for mined transactions older than the backend's transaction-by-hash retention window on some archive nodes (observed on QuikNode Base, now the Base backend). Such transactions are still fully retained and reachable via eth_getBlockByHash and eth_getTransactionReceipt, but blockbook reported them as ErrTxNotFound, failing both production lookups and the base=main/rpc/GetTransaction integration test. Fall back to reconstructing the transaction from its receipt (for the block hash) plus the full block body when eth_getTransactionByHash returns null. Genuinely unknown transactions still return ErrTxNotFound, and the primary path is unchanged for backends that retain the full index. Fixes #1609 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
recoverMinedTransaction reconstructed a pruned-index transaction by fetching the entire block body (eth_getBlockByHash, fullTxs=true) and scanning it for the hash. On endpoints that call GetTransaction per tx (address/xpub/block/balancehistory), a cold-cache page of transactions from the pruned window fetched and re-parsed the full block body once per tx - the same ~592 KB block up to 1000x on a block page. Fetch the transaction directly by its receipt's (blockHash, transactionIndex) via eth_getTransactionByBlockHashAndIndex - an O(1), ~900x smaller lookup - and reuse the already-fetched receipt for EthTxToTx instead of fetching it a second time. Verified against the QuikNode Base archive backend: the by-index result is byte-identical to the block-body scan across transactions 64-926 days old. Fall back to the block-body scan when the positional lookup is unavailable, so the change is strictly non-regressive for any backend, and give each recovery RPC its own timeout budget. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add <network>_ALLOWED_EVM_CALL_METHODS, a comma-separated list of 4-byte method selectors that permits websocket rpcCall requests to any address when the calldata starts with an allowed selector (e.g. 0xdd62ed3e for ERC-20 allowance). Complements ALLOWED_RPC_CALL_TO: a call passes when either its target address or its selector is allowed; with neither set rpcCall stays unrestricted. Malformed calldata never matches (full hex decode, fail closed) and malformed or empty selector config fails startup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Turn ALLOWED_RPC_CALL_TO and ALLOWED_EVM_CALL_METHODS into runtime settings: the environment variables remain the startup defaults, while an override written through the authenticated internal API GET/POST/DELETE /admin/runtime-settings/<KEY> is persisted in RocksDB (own cfDefault rows, independent of the periodically stored internalState blob and invisible to older versions), survives restarts, takes precedence over the environment and takes effect on websocket rpcCall immediately, without a restart. The live allowlists are an immutable snapshot behind an atomic pointer on InternalState, so the rpcCall hot path stays lock-free and both servers (public websocket, internal admin) share one view regardless of which of them a deployment runs. Writes are strictly ordered: validate (400 on invalid input) -> store to DB (500 and untouched live state on failure) -> publish snapshot -> log the change with old/new value, source and client address. POST of an exactly empty value explicitly unconfigures a dimension; whitespace- or separator-only values are rejected so a botched input cannot silently un-restrict rpcCall; DELETE reverts to the environment default and validates it first so a malformed environment cannot brick the next restart. The TO-list parser is unified with the selector parser: entries are trimmed, empty entries skipped and a set-but-empty value fails startup (previously "," allowlisted the empty string, which matched rpcCall requests with an empty to field). A minimal /admin/runtime-settings page lists the current values and their sources with curl hints. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Avoids allocating a decoded copy of the full calldata (up to 4 MiB) on every rpcCall allowlist check. The full hex encoding is still validated byte-by-byte to preserve the existing fail-closed behavior for malformed calldata.
Addresses the API unification review on PR #1597: - POST /admin/contract-info/ returns a typed {"updated":N} object instead of a pre-serialized string that jsonHandler double-encoded on the wire; a malformed body is now a 400 (public APIError) rather than a 500 - apiContractInfo dispatches on method like apiRuntimeSetting: POST/PUT write the collection path only (an address segment is rejected instead of silently ignored), unknown methods are a 400 instead of falling into the GET path - new DELETE /admin/contract-info/<address> purges cached contract metadata (DB row + LRU entry) so it is re-fetched from the backend on the next read; idempotent, missing row reports deleted:false - shared urlPathSegment helper unifies path-segment extraction; contract addresses get no case normalization (the chain parser is the authority), runtime-setting keys keep their uppercasing at the call site - runtime-settings routes and initRpcCallAllowlists move out of the ChainEthereumType guard (the mechanism is chain-generic and the init already runs unconditionally in NewWebsocketServer); contract-info and internal-data-errors stay EVM-gated, nav links follow the same split Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Smaller items from the PR #1597 review: - lowercase the "invalid EVM call method selector" parse error to match the sibling parser's message convention - trim environment values wherever they are resolved (startup and the DELETE fallback), so stray whitespace from an env file never surfaces in values; a set but whitespace-only env value is a configuration error rather than unset — treating it as unset would silently un-restrict rpcCall - document the differing To/Methods map key formats on RpcCallAllowlists so a future allowlist dimension picks a convention deliberately - runtimeSettingStore gains GetRuntimeSetting; resolveRuntimeSetting reads through the interface instead of requiring *db.RocksDB Admin runtime-settings page gets an inline editing UI (edit/save/delete per setting, replacing the curl-only instructions) and the page/nav/ routes are consistently available on all chain types. Hardening found by review of the new DELETE /admin/contract-info/: - bump protocolGen before the cache purge in DeleteContractInfoForAddress so a concurrent GetContractInfo cannot re-insert the deleted row into the LRU (same idiom as SetErcProtocol); without it the purge could be silently undone until eviction or restart - DELETE returns (and logs) the purged record: the backend re-fetch restores only name/symbol/decimals, not the sync-owned createdInBlock/destructedInBlock, so the response gives the operator a POST-back restore path and the docs now state exactly what is discarded - a failed save in the admin UI keeps the row in edit state instead of stranding an orphaned input, and error responses are unwrapped to their message Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Supports the bbctl counterpart (trezor/bbctl#10) of the runtime-settings admin API in a two-replica deployment where each replica owns its DB: - GET /admin/runtime-settings/ (bare collection path) returns every setting with its effective value and source as a JSON array, so a management tool reads the whole state per replica in one request; other methods on the bare path stay a 400 - initRpcCallAllowlists warns at startup when a stored override shadows a different (or malformed) environment value, making env/DB drift — a replica that missed an admin update, or an env change rolled out while an override exists — visible; the shadowed value is only compared, never parsed, so drift cannot fail a restart - docs/env.md explains the deployment roles of the two sources: the env var is the deploy-managed baseline that survives a replica resync (the DB is wiped, and unset allowlists would silently un-restrict rpcCall), the stored override is the runtime layer that survives restarts until the next deploy ships an updated environment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GET /admin/contract-info/ (bare collection path) lists the stored
records, mirroring the runtime-settings list convention. Unlike runtime
settings the collection is unbounded — sync stores a record per contract
creation, millions on a busy chain — so the list is paginated:
?limit=<1..10000> (default 1000) with a from cursor, and the response
{"contracts":[...],"next":"<address>"} carries the from of the next
page. ListContractInfos iterates cfContracts in address-descriptor
order (the CF holds only plain addrDesc keys; protocol rows live in
their own CF).
The former 400 for GET of the bare path moves to the list; DELETE and
POST semantics on the bare path are unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Post-deploy step in the e2e-tests job exercising the admin API (/admin/runtime-settings/, /admin/contract-info/) of the deployed base_archive dev instance, leaving its database and live rpcCall policy exactly as found — at the end of a run and at every instant during it: - runtime settings use a zero-policy-impact roundtrip: the POSTed value is always the current effective value (or the explicit-empty override when unset), so the parsed allowlist never changes; the final DELETE restores the original source; db-sourced settings are only rewritten in place, since their env fallback is unknown from outside - the contract roundtrip uses a reserved never-real address, pre-cleans it (healing a crashed run) and deletes it at the end; the cleanup and restore flags are raised before the POSTs, so a response lost after a server-side commit still gets cleaned up - negative tests: 401 without credentials (503 = admin disabled fails loudly), unknown key, invalid values proven to change nothing, POST to an address path, unsupported method, list pagination limits - target URL derives from BB_DEV_API_URL_HTTP_<coin> plus the coin config's ports.blockbook_internal (BB_ADMIN_E2E_URL overrides for local runs); the coin gate compares against coins_csv deploy aliases (test_coins_csv carries test names and would never match); skips are annotated (::warning:: for a missing BB_RUNTIME_ENV, ::notice:: for coin-not-deployed) so a dead signal cannot look like a pass - credentials are parsed from BB_ENV inside the script (surrounding quotes stripped like systemd EnvironmentFile) and never printed; transient network errors are retried once, then fail with a clean pipeline error; helpers reuse runner.py and wait_for_sync.py Verified live against blockbook-dev3: two consecutive runs green via the derived URL, instance state identical before and after. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The BB_DEV_API_URL_HTTP_<coin> repo variables follow the tests.json test-name convention (BB_DEV_API_URL_HTTP_base for base_archive), so resolving them by the deploy alias failed with "missing BB_DEV_API_URL_HTTP_base_archive". Map the deploy alias through runner.load_test_coin_name (the same helper deploy_plan.py uses) before the lookup; the coin gate keeps comparing deploy aliases. Unit test pins the mapping against the real base_archive.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The paginated contract listing discarded the error from GetAddressesFromAddrDesc and fell through to return next="" for a boundary row whose key did not decode to an address. That silent truncation is indistinguishable from a completed listing and drops the remaining rows; in-page rows with no decoded address were likewise returned with an empty Contract. A corrupt key now returns an error, matching the unpackContractInfo error handling in the same loop. Unreachable for well-formed EVM/Tron contract descriptors, but a visible failure beats a confusing partial page. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(tron-testnet): upgrade backend to 4.8.2 * fix(tron-testnet): path to the .jar backend file * fix(tron-testnet): update backend version
…esilliency (#1582) * feat(fiat-api): add cap limits for fiat rate endpoints * feat(evm-contract): verify contract logs to avoid possible OOM * feat(websocket): add lower concurrency cap on getMempoolFilters * fix(fiat-api): apply currency count cap to raw selector list length The MaxFiatRatesCurrencies check fired only when a unique normalized code was appended, so an all-duplicate list bypassed the cap while still costing a full trim/lowercase/dedup pass per element. Reject over-long lists up front so the normalization work itself is bounded by the cap; this also makes the in-loop unique-count check unreachable, so drop it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(fiat-api): normalize currency selectors once at entry points setFiatRateToBalanceHistories re-normalized a slice that both of its callers (GetBalanceHistory, GetXpubBalanceHistory) had already passed through normalizeFiatCurrencies, and getFiatRatesResult re-lowercased selectors already lowercased by its callers. Drop the redundant passes and document the pre-normalized precondition instead. Normalization was the only error path of setFiatRateToBalanceHistories, so it no longer returns an error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refactor(evm-contract): drop unreachable ERC-1155 count overflow guard The endValues <= len(data) check already bounds countValues to len(data)/evmWordHex, far below maxInt, so the truncation guard on the int conversion could never fire (the load-bearing overflow check lives in erc1155BatchArrayEnd). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(websocket): clarify scope of getMempoolFilters response cap The slot is deliberately held from before the handler computes the response until it is written to the websocket (or drained on close): the cap exists to bound peak memory in computed-but-unwritten filter responses, so it covers compute, queueing, and write together rather than compute concurrency alone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(eth-parsing): skip unparseable log with a warning and continue with next logs * test(websocket): cover mempool filter slot release after successful write Adds TestWebsocketOutputLoopReleasesMempoolFilterSlotAfterWrite, which drives a release-bearing WsRes through outputLoop over a live websocket connection and asserts the mempool-filters slot is freed after WriteJSON. This pins the primary release path (outputLoop's c.finalize(m)) that the existing semaphore and drain-on-close tests do not exercise; without it, removing that call would leak a slot on every successful response and go undetected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
getEnsRecord read an attacker-controlled 32-byte word from an ENS NameRegistered log and computed a slice end as 194+64+(int(c)<<1). For c in [0x7FFFFFFFFFFFFF7F, 0x7FFFFFFFFFFFFFFF] the signed left shift wraps and de lands in [0, 256], slipping the `de < 0` guard, so l.Data[194+64:de] became a low>high slice expression and panicked. The panic runs in the GetBlock log-processing goroutine, which has no recover(), so it crashes the whole process before the block is committed -> deterministic crash-loop on restart. - getEnsRecord: guard the slice start (de < 194+64) instead of de < 0. - processParam: same fix (de < d) for the sibling data[d:de] slice (already under ParseInputData's recover(); correctness only). - GetBlock: recover() the log goroutine as defense-in-depth, returning an error instead of letting the process die (adds runtime/debug). - tests: regression cases for both overflow paths (de=0, de=256, and the processParam variant), verified to panic without the fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tart in getEnsRecord Address two review comments: - Add d+64 < d overflow guard at line 159 in processParam, same class as the existing de < d fix. - Replace magic 194+64 with a local const nameStart in getEnsRecord.
TestGetTickersForTimestamps_ConcurrentReadersAndWriters and TestLogTickersInfo_ConcurrentWithCacheWriters wait `waitTimeout` for their reader/writer/logger goroutines to drain after `stop` is closed. The deadline was 3s. That is only a deadlock safety-net, but it is too tight on a CPU-starved CI runner (the Unit Tests job compiles RocksDB in parallel), so `wg.Wait()` occasionally misses it and the test fails with "concurrent ... did not finish in time". Raise it to 30s: still fails fast on a genuine deadlock, but tolerant of scheduling delays under load. Test-only change; the concurrency exercise (driven by testDuration) is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GetTransactionFromBchainTx resolved each input's previous output by calling GetTxAddresses, which unpacks an entire TxAddresses record (all inputs and outputs) just to read a single output. For transactions with many inputs that reference large previous transactions this allocates on the order of inputs x record size, driving heap usage up sharply on high-fan-in addresses. Add db.GetTxAddressesOutput, which returns a single Outputs[vout] by skipping over the height, inputs and preceding outputs by length without allocating them, and use it in the vin-resolution loop. Allocation now scales with the number of inputs rather than inputs x record size. Adds TestGetTxAddressesOutput (both index modes) and BenchmarkTxInputResolution covering the input fan-in scaling.
…on self-hosted runners The actions/setup-node built-in npm cache saves ~/.npm with paths relative to the workspace. On runners where the home directory lives outside the workspace tree, tar cannot restore these paths. The explicit actions/cache for tests/openapi/node_modules already handles caching and skips npm ci on hit, making the global npm cache redundant.
actions/checkout v4.3.1 -> v7.0.0 actions/cache v4.3.0 -> v6.1.0 actions/setup-node v4.4.0 -> v6.4.0 actions/upload-artifact v4.6.2 -> v7.0.1 All four actions were targeting Node.js 20, which is now deprecated on GitHub Actions runners running Node.js 24. The new major versions bundle a compatible runtime.
Instead of setup-node's built-in cache:npm (which saves ~/.npm with paths relative to the workspace, causing tar failures when home is outside the workspace tree), cache a local .npm-cache directory inside the workspace via explicit actions/cache, and pass --cache .npm-cache to npm ci. On node_modules cache miss, the local cache avoids re-downloading packages from the registry. The restore-keys fallback preserves partial cache benefit even across lockfile changes.
On node_modules cache hit, npm ci is skipped and .npm-cache was never created, causing the post-job cache save to warn: Path Validation Error: Path(s) specified for caching do not exist Pre-create the directory with mkdir -p so the path always exists for the save phase. On cache hit the directory is empty and the save is a no-op.
…ng chains The getSampleIndexedHeight probe searches from bestHeight-2 backward for an indexed block hash. At 12 blocks (txSearchWindow), chains with slower indexers (ethereum-classic on core-geth/blockbook-dev2) sometimes have zero indexed blocks in the window, causing flaky failures. 20 blocks covers ~4.5 minutes of ETC chain history, ample for any realistic indexing lag. The loop stops on the first hit, so healthy runs are unaffected.
* fix(ws): cap subscribeFiatRates tokens array to prevent DoS The subscribeFiatRates handler stored the request's tokens slice whole for the connection lifetime and re-iterated it on every fiat-rate broadcast (per subscribed currency) while holding the global fiatRatesSubscriptionsLock. It was the only WebSocket list parameter without a length cap: a single message up to the 4 MiB read limit could carry ~10^6 token strings, enabling amplified heap retention and periodic global-lock contention that stalls fiat-rate subscription handling. Reject requests whose tokens array exceeds maxWebsocketSubscribeFiatRatesTokens (1000) with a public API error, matching the existing caps on addresses, currencies, timestamps, and estimateFee blocks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore(websocket): cleanup comment --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(eth): prevent duplicate ERC-721 holdings * chore(contrib-script): move the erc-721 holdings check to a gist * refactor(eth): address PR review comments on NFT self-transfer handling - derive the NFT self-transfer invariant from a shared isNFTSelfTransfer helper on both the connect and the disconnect side so they cannot drift - document that addToContract callers pass mutateHoldings=false only for ERC721 self-transfers - add a ConnectBlock -> DisconnectBlock round-trip test verifying that an ERC721 self-transfer keeps holdings untouched and tx counts balanced Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…urable The cap on concurrently executing requests per connection is a hard-coded const (48). Internal batch clients that pipeline many requests on a single connection (e.g. bulk getAccountInfo address scans by an exchange backend) trip it and get disconnected mid-batch, with no way to tune the limit -- unlike the adjacent websocket message-rate and getAccountInfo limits, which are already env-configurable. Read the limit from <NETWORK>_WS_PENDING_REQUESTS_LIMIT following the existing WS limiter env pattern; the default stays 48 and 0 disables the cap (the server-wide work limit remains as a backstop). Document the variable in docs/env.md and cover the override with a test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.