Skip to content

Add support for dual reads in mongobetween#48

Closed
kounat wants to merge 9 commits intocoinbase:masterfrom
kounat:dual-reads
Closed

Add support for dual reads in mongobetween#48
kounat wants to merge 9 commits intocoinbase:masterfrom
kounat:dual-reads

Conversation

@kounat
Copy link
Contributor

@kounat kounat commented May 11, 2022

This PR adds support for dual reads, which allows us to make reads to a migrated cluster and verify performance/correctness in a way that is invisible to the application.

The second round trip is made after the socket write and the client should see no change in latency with dual reads enabled.

func MustOpMsgCursorSection(op Operation) []byte {
opmsg, _ := op.(*opMsg)
section := opmsg.sections[0].(*opMsgSectionSingle)
cursorDoc := section.msg.Lookup("cursor").Document()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just making a note so I remember where I'm at. I was starting my stress testing and saw some panics from this line of code:

{"level":"error","ts":1653593508.0804174,"caller":"proxy/connection.go:35","message":"Connection crashed","remote_address":"@","panic":"Call of bsoncore.Value.Document on invalid type","stack":"goroutine 267 [running]:\nruntime/debug.Stack()\n\t/usr/local/Cellar/go/1.17.2/libexec/src/runtime/debug/stack.go:24 +0x65\ngithub.com/coinbase/mongobetween/proxy.handleConnection.func1()\n\t/Users/sean.hurley/workspace/mongobetween/proxy/connection.go:35 +0xc7\npanic({0xbd8e00, 0xc0010418d8})\n\t/usr/local/Cellar/go/1.17.2/libexec/src/runtime/panic.go:1038 +0x215\ngo.mongodb.org/mongo-driver/x/bsonx/bsoncore.Value.Document({0x75, {0x0, 0xc, 0xc00192f6e8}})\n\t/Users/sean.hurley/go/pkg/mod/go.mongodb.org/mongo-driver@v1.8.0/x/bsonx/bsoncore/value.go:443 +0x15f\ngithub.com/coinbase/mongobetween/mongo.MustOpMsgCursorSection({0xe38c90, 0xc000f72bd0})\n\t/Users/sean.hurley/workspace/mongobetween/mongo/operations.go:363 +0xaa\ngithub.com/coinbase/mongobetween/proxy.(*connection).roundTripWithDualCursor(0xc00192fed8, 0xc000b9cab0, 0xc000b9cd80, 0x0, {0xc0006354a0, 0x6, 0xa})\n\t/Users/sean.hurley/workspace/mongobetween/proxy/connection.go:229 +0x874\ngithub.com/coinbase/mongobetween/proxy.(*connection).handleMessage(0xc00192fed8)\n\t/Users/sean.hurley/workspace/mongobetween/proxy/connection.go:136 +0xc25\ngithub.com/coinbase/mongobetween/proxy.(*connection).processMessages(0xc00192fed8)\n\t/Users/sean.hurley/workspace/mongobetween/proxy/connection.go:55 +0x3d\ngithub.com/coinbase/mongobetween/proxy.handleConnection(0x0, 0xc6626f, {0xc00007a540, 0x0}, {0xe37700, 0xc00000e180}, 0x0, 0x0, 0x0)\n\t/Users/sean.hurley/workspace/mongobetween/proxy/connection.go:50 +0xd1\ngithub.com/coinbase/mongobetween/proxy.(*Proxy).accept.func2()\n\t/Users/sean.hurley/workspace/mongobetween/proxy/proxy.go:165 +0xae\ncreated by github.com/coinbase/mongobetween/proxy.(*Proxy).accept\n\t/Users/sean.hurley/workspace/mongobetween/proxy/proxy.go:163 +0x368\n"}

Will need to dig in more as to why that's happening (Looks like we're assuming ALL reads have a cursor in the response, but might not be the case

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a little more logging to see what the op looked like when this error is happening:

{ OpMsg flags: 0, sections: [{ SectionSingle msg: {"ok": {"$numberDouble":"0.0"},"errmsg": "No keys found for HMAC that is valid for time: { ts: Timestamp(1653595159, 1562) } with id: 7102116512885899268","code": {"$numberInt":"211"},"codeName": "KeyNotFound","$clusterTime": {"clusterTime": {"$timestamp":{"t":"1653595159","i":"1561"}},"signature": {"hash": {"$binary":{"base64":"Nthr9PHRWcpILtOPqyful0plsf8=","subType":"00"}},"keyId"
: {"$numberLong":"7102116560130539524"}}},"operationTime": {"$timestamp":{"t":"1653595159","i":"1561"}}} }], checksum: 0 }

Will dig in more later

@kounat
Copy link
Contributor Author

kounat commented May 31, 2022

Closing in favor of #51

@kounat kounat closed this May 31, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants