Repository navigation
py-multihash ↔ go-multihash Feature Parity Analysis #1361
sumanjeet0012
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Feature Parity Analysis
Table of Contents
Executive Summary
The Python implementation (
py-multihash) has made significant progress in v3.0.0 with rich features likeMultihashSet, JSON serialization, stream read/write, SHAKE support, truncation, and an extensiveFuncenum covering 150+ hash codes. However, several important areas remain incomplete compared to the Go reference implementation:sum_stream)Visit(),ForEach(),Len()RegisterVariableSize(),DefaultLengthsmapcore+register/*packagesoptspackageCast(),MHFromBytes(),NewReader()/NewWriter()Architecture Comparison
Go Architecture (Modular)
Key design principle: The
corepackage has zero transitive dependencies. Hash implementations live in separateregister/*packages that users import only if needed (via side-effect imports_ "..."). This keeps binary sizes small.Python Architecture (Monolithic)
Key difference: All hash implementations (blake3, mmh3, hashlib) are loaded eagerly at import time. There is no lazy-loading or opt-in registration.
Feature-by-Feature Gap Analysis
3.1 Core Types & Data Structures
Multihashtype (byte slice / tuple)Multihash []byteMultihashnamedtupleDecodedMultihashstructDecodedMultihash{Code, Name, Length, Digest}Multihashnamedtuple (same fields)HexString()/String()Multihashto_hex_string()standalone +Multihash.encode('hex')B58String()Multihashto_b58_string()standalone +Multihash.encode('base58')FromHexString()FromHexString(s string)from_hex_string(s)FromB58String()FromB58String(s string)from_b58_string(s)Cast(buf []byte)MultihashtypeMHFromBytes(buf []byte)(bytesRead, Multihash, err)for parsing from a larger bufferis_valid()Cast)is_valid(multihash)get_prefix()[:2])get_prefix(multihash)is_app_code()is_app_code(code)coerce_code()coerce_code(hash_fn)to_json()/from_json()verify()Multihash.verify(data)3.2 Encoding & Decoding
Encode(buf, code)encode(digest, code, length)EncodeName(buf, name)encode(digest, name)Decode(buf)*DecodedMultihashdecode(multihash)returnsMultihashgo-varintlibraryvarintlibraryget_prefix()hardcodes[:2]which breaks for multi-byte varintsget_prefix()issue;decode()should handle butget_prefix()won't3.3 Hash Computation (Sum)
Sum(data, code, length)length=-1for defaultsum(data, code, length)SumStream(reader, code, length)sum_stream(stream, code, length)sum[:length]TruncationErrorlength == len(data)ErrLenTooLargeon oversized requestTruncationError(different semantics)3.4 Stream I/O
Multihash.read(stream)ReadMultihash)Multihash.read(stream)Multihash.write(stream)WriteMultihash)Multihash.write(stream)Readerinterface /NewReader()io.ReaderwithReadMultihash()Writerinterface /NewWriter()io.WriterwithWriteMultihash()ReadMultihash()on readerMultihash.read()WriteMultihash()on writerMultihash.write()3.5 MultihashSet
SetMultihashSetNewSet()/ constructorNewSet()MultihashSet(iterable)Add(m)Add()+add()Has(m)Has()+__contains__Remove(m)Remove()+remove()+discard()All()[]Multihashlist[Multihash]Len()__len__()Visit(m)→ boolTrueif newForEach(f func(Multihash) error)TypeErrorchecks3.6 Registry System
coreFuncRegRegister(code, factory)register(code, name, hash_name, hash_new)RegisterVariableSize(code, factory(sizeHint))GetHasher(code)hash.Hashhash_from_func(func)GetVariableHasher(code, sizeHint)DefaultLengthsmapmap[uint64]intHASH_LENGTHSin constants but not in registry)Unregister(code)unregister(code)for app codesReset()reset()func_from_hash(obj)register/blake2,register/sha3, etc.funcs.pycore.init()FuncReg.reset()3.7 Hash Function Coverage
Hash functions available in BOTH implementations:
Hash functions in Python but NOT (fully) in Go:
Hash functions in Go but NOT in Python:
Variable-length BLAKE3 gap:
ErrLenTooLargefor > 128murmur3 naming & behavior inconsistency:
"murmur3-x64-64""murmur3-128""murmur3-32"(4 bytes)spaolacci/murmur3mmh33.8 CLI Tool & Options Package
multihash)multihash file.txtcat file | multihash--check <hash>-a sha2-256-e hex/base58/base64/raw-l 256(bits)-qopts.Optionsstructopts.SetupFlags()opts.Decode(encoding, str)opts.Encode(encoding, mh)opts.Check(reader, multihash)3.9 Error Handling
ErrUnknownCodeUnsupportedCodeErrorErrTooShortValueError("multihash must be greater than 3 bytes")ErrTooLongErrLenNotSupportedErrInvalidMultihashInvalidMultihashErrorErrVarintBufferShortValueError("Invalid varint provided")ErrVarintTooLongValueError("Invalid varint provided")ErrInconsistentLen(typed struct)ValueError("Inconsistent multihash length")ErrSumNotSupportedHashComputationErrorErrLenTooLargeTruncationErrorMultihashError(base)3.10 Testing & Benchmarks
TestSpecTestSpecVectorsmustNotAllocateMoreBenchmarkEncode,BenchmarkDecode,BenchmarkCast,BenchmarkSum,BenchmarkBlake2B,BenchmarkSumAllLargeTestSmallerLengthHashIDFeature Matrix Summary
Cast()functionMHFromBytes()functionNewReader()/NewWriter()wrappersMultihashSet.Visit()MultihashSet.ForEach()MultihashSet.Len()__len__)RegisterVariableSize()GetVariableHasher()DefaultLengthsmap in registryget_prefix()fix for multi-byte varintsmultihashbinary)optspackagePhased Roadmap
Phase 1 – Critical Bug Fixes & Core API Gaps
Goal: Fix bugs that break correctness and add missing Go-style core APIs.
Estimated effort: 1–2 weeks
1.1 Fix⚠️ BUG
get_prefix()for multi-byte varint codesFix: Use varint decoding to determine the actual prefix length (code varint + length varint).
1.2 Add
Cast()functionGo-compatible buffer-to-Multihash casting with validation:
1.3 Add
MHFromBytes()function1.4 Register Keccak-256 and Keccak-512 hashers
These are defined in the
Funcenum but have no registered hash implementation. Add wrappers usingpysha3orpycryptodome:1.5 Add Identity hash length validation in
sum()Go enforces that identity hash length must equal data length:
1.6 Add
NewReader()/NewWriter()wrapper classesPhase 2 – Registry Parity & Variable-Size Hashes
Goal: Bring the registry system to full parity with Go's
core/registry.go.Estimated effort: 1–2 weeks
2.1 Add
RegisterVariableSize()toFuncReg2.2 Add
GetVariableHasher()toFuncReg2.3 Add
DefaultLengthsmap toFuncRegPopulated automatically when functions are registered.
2.4 Variable-length BLAKE3 support
Update
Blake3Hashto accept alengthparameter (32–128 bytes) and integrate with the variable-size registry:Register with
RegisterVariableSize:2.5 Add Keccak-224 and Keccak-384 registration (if library available)
Using
pycryptodome'sCrypto.Hash.keccak:Phase 3 – Modular Architecture & Spec Compliance
Goal: Restructure the package for modularity and add spec compliance verification.
Estimated effort: 2–3 weeks
3.1 Create
multihash.coresub-packageExtract constants and registry into a minimal sub-package with zero external dependencies:
3.2 Add lazy-loading for optional hash libraries
Currently
blake3andmmh3are hard dependencies. Move them to optional extras:3.3 Add spec compliance tests
Test 1: Validate
HASH_CODESagainst the official multicodec table.csv:Test 2: Run official test vectors:
3.4 Add
version.json{ "version": "3.0.0" }Phase 4 – CLI Tool & Options Package
Goal: Provide a CLI tool and options-parsing package matching Go's
multihashbinary.Estimated effort: 1–2 weeks
4.1 Create
multihash.optsmodule4.2 Create CLI entry point
Register as console script in
pyproject.toml:4.3 CLI Features to implement
multihash file.txtcat file | multihash-a sha2-256,-a blake3-e hex,-e base58,-e base64-l 128(in bits)--check <multihash>-q--helpPhase 5 – Testing, Benchmarks & Polish
Goal: Comprehensive test coverage, benchmarks, and API polish.
Estimated effort: 1–2 weeks
5.1 Expand Sum test vectors
Port all 40+ Go test cases from
sum_test.goto Python:5.2 Add varint edge-case tests
5.3 Add identity hash edge-case tests
length == len(data)→ successlength < len(data)→ errorlength = -1→ success5.4 Add evil reader test
Test that
Multihash.read()works correctly when the stream returns data one byte at a time.5.5 Add benchmark suite
5.6 Complete
MultihashSetGo API parity5.7 Add error type parity
Add missing specific error types:
5.8 Add deprecated aliases (backward compat with older Python multihash libs)
Appendix – Naming Inconsistencies Between Implementations
These naming differences exist between Go and Python and should be documented (not necessarily "fixed", as changing them would be a breaking change):
murmur3-x64-64(8 bytes)murmur3-128(16 bytes)murmur3-128(32 bytes!)murmur3-32(4 bytes)murmur3-32IDENTITY+ deprecatedIDFunc.identityidSHA3 = SHA3_512sha3-512MURMUR3 = MURMUR3X64_64Summary Timeline
get_prefix(), addCast(),MHFromBytes(), Keccak registration, Reader/Writer wrappersRegisterVariableSize(),DefaultLengths, variable-length BLAKE3, Keccak-224/384coresub-package, lazy-loading, spec compliance tests, optional depsmultihashCLI tool,optsmodule, encoding utilitiesVisit()/ForEach(), error parityTotal estimated effort: 6–11 weeks
Each phase is independently shippable and provides incremental value. Phase 1 is the highest priority as it fixes correctness bugs. Phase 3 is the most impactful for long-term maintainability.
All reactions