weblist comes back blank on OSX when binary referenced in profile exists

[tbg]

Repro (needs to be on OSX, in pprof repo)

$ go test -o ./profile/foo -c -memprofile=mem.pb.gz ./profile/
$ (cd profile/ && ./foo -test.memprofile ../mem.pb.gz)
PASS
$ go tool pprof -http :6060 mem.pb.gz

In the browser window, navigate to the flame graph and click "Show source in new tab":

A completely empty source listing tab opens.

I spent way too much time digging into this today and found that this is related to whether the file referenced in the profile exists. That's right, you rm profile/foo and refresh the browser window, it will just work.

I didn't just try that out randomly but was motivated to try this after spending a few hours with the debugger attached.

When the file exists, we hit this code:

pprof/internal/binutils/addr2liner_nm.go

Lines 63 to 72 in a229014

	if cmd == "" {
	cmd = defaultNM
	}
	var b bytes.Buffer
	c := exec.Command(cmd, "--numeric-sort", "--print-size", "--format=posix", file)
	c.Stdout = &b
	if err := c.Run(); err != nil {
	return nil, err
	}
	return parseAddr2LinerNM(base, &b)

and (*sourcePrinter).addStack will get (from the addr2LinerNM) plugin.Frames that have zero File and Line, i.e. the only thing in them is Func. Not only that, Func will also have an errant underscore prefix, which is an OSX thing (https://go-review.googlesource.com/c/go/+/196217). So basically it can never successfully look up which file a Frame is from. So at best we can expect a listing that doesn't reference the files.

But we don't even get that! We see why in this snippet as well. sp.interest is keyed by function names as stored in the profile, which don't have the underscore. But we look up with the underscore. So it never matches.

I hacked around this with a diff that also adds the underscore version into sp.interest. Then we get something like this (this is from when I had not ported my repro to use the pprof repo):

Meanwhile, interactive pprof's list command works flawlessly. When I looked at the source I realized why: list doesn't seem to share much code (if any at all) with weblist.

weblist is obviously much more powerful, so it would be good if it worked out of the box on OSX. Moving the file out of the way gives the basic listing, but disables the advanced functionality that's tied to having the binary present.

I have some more "meandering" (write-as-you-go) notes at https://gist.github.com/tbg/ff8de01920551be1a2193a2fd3ffd6b8, I don't think they're adding much but linking just in case.

[tbg]

I assume this will be fixed when #477 is.

[tbg]

Just confirming that this was not fixed by #924, which came out of #477.

refresh after rm the binary:

[tbg]

I take it back - had updated the pprof binary incorrectly. Thank you @alk!

[tbg]

Sad to report that it still doesn't always work. I hit another blank page when navigating to source, which rendered only after removing the binary referenced in the profile.

[Denvormine]

I'm facing this same issue on Ubuntu. list works, webUI just shows a blank page or a page with single entry like runtime.gopark. Looks like it happens only when I'm profiling applications that are running locally.
EDIT: I guess 'whether the file referenced in the profile exists' explains "only locally" part.
Thank you for your issue, I'm now able to finally see lines of code again.

[alk]

Any similarly looking issues, but on Linux or any other decent (read: ELF) system should IMHO be a separate ticket. I advice you to create new ticket with as many details as possible. And feel free to CC me. I am not a member of pprof team, but I have some matching interests and I'll be happy to look at your case.