GO_COV issues with resolving paths

[egonelbre]

coverage-model/src/main/java/edu/hm/hafner/coverage/parser/GoCovParser.java

Lines 43 to 52 in 2833a79

	private static final Pattern LINE_PATTERN = Pattern.compile(
	"(?:(?<org>[^/\\\\:]+\\.[^/\\\\:]+)[/\\\\])?"
	+ "(?<project>[^/\\\\:]+)[/\\\\]"
	+ "(?<module>[^/\\\\:]+)[/\\\\]"
	+ "(?<package>.*[/\\\\]?.*)[/\\\\]"
	+ "(?<file>[^/\\\\:]+):"
	+ "(?<lineStart>\\d+)\\.(?<columnStart>\\d+),"
	+ "(?<lineEnd>\\d+)\\.(?<columnEnd>\\d+)\\s*"
	+ "(?<statements>\\d+)\\s*"
	+ "(?<executions>\\d+)");

In coverage profile output the format is "<package>/<filename>:<linestart>". There's no standard structure how modules and orgs are related to package path.

For example having a report for example.com/main.go:1 is entirely plausible. I created a small example here https://github.com/egonelbre/exp/blob/main/coverage_example/coverage.out.

Since reimplementing module path discovery logic inside coverage parser seems rather a huge undertaking. Shelling out to go is probably not a good idea either, maybe there's a way to pass in arguments to the parser on how to convert package names to paths. e.g. passing in prefixes and how they should be replaced with paths:

example.com=./
example.com/submodule=/tmp/submodule
github.com/google/protobuf=./vendor/protobuf

---

For context, I finally started migrate from corberatura plugin to coverage plugin and I'm getting errors such as:

[Coverage] [-ERROR-] - Source file 'xyz/abc.go' not found

[uhafner]

Does the option sourceDirectories work in this case?

[egonelbre]

I wasn't able to get it to work. It seems the current process ends up trimming some of the directories that are necessary for determining where the file is located.

For example, if you have a module "example.com/project" with files "internal/alpha.go", "public/beta.go" -- then the full paths would be "example.com/project/internal/alpha.go" and "example.com/project/public/beta.go". The current parser ends up with "alpha.go" and "beta.go" losing "internal" and "public" parts of the path.

Similarly, you could have multiple modules that are located in different folders; although, I do not have this problem.

[uhafner]

I see. I'm not sure what is the best way to fix the problem:

1. Let the authors of GO_COV know that the report format should be improved so that other tools can better detect the file names
2. Improve the parser, so that the input format is better split into modules and file names
3. Add a mapping table

What do you think?

[egonelbre]

Changing Go coverprofile format seems unlikely, because it fits how Go works with tooling. It's also somewhat a legacy format so it feels unlikely that they would like to modify it. There is a new format as well, but that's a binary format. The binary is not that trivial to parse though and is not really meant for external consumption.

To improve the parser, the full logic to find filepath from the modules seems infeasible, because some of the external dependencies might be in some "/tmp/gomod" folder. Something simpler could work such that the plugin takes all the go.mod files as input, and then figures out how to translate. Note, there can be multiple go.mod files which may not align specifically with the folder structure, but all the files within the module should be fine. I'll write the go.mod approach below in more detail.

Adding a mapping table seems the easiest logic-wise and most configurable. However, from usability it's slightly worse than go.mod approach.

---

I haven't written Java in a while, so here's Go code how to do rough package filename translation using go.mod files.

https://github.com/egonelbre/exp/blob/main/coverage_example/parse/main.go

Details

type Module struct {
	Prefix string
	Dir    string
}

// See https://go.dev/ref/mod#go-mod-file for the spec.
func ParseModule(path string) (Module, error) {
	data, err := os.ReadFile(path)
	if err != nil {
		return Module{}, err
	}

	for _, line := range strings.Split(string(data), "\n") {
		line = strings.TrimSpace(line)
		// is it a module defintion line?
		if strings.HasPrefix(line, "module ") {
			// trim the prefix
			modulename := strings.TrimPrefix(line, "module ")
			// cut any trailing comments, if there are any
			modulename, _, _ = strings.Cut(modulename, "//")
			modulename = strings.TrimSpace(modulename)

			return Module{
				Prefix: modulename + "/",
				Dir:    filepath.Dir(path),
			}, nil
		}
	}

	return Module{}, errors.New("invalid module")
}

func TranslatePath(pkgpath string, modules []Module) (path string, found bool) {
	for _, module := range modules {
		if strings.HasPrefix(pkgpath, module.Prefix) {
			f := filepath.Join(module.Dir, strings.TrimPrefix(pkgpath, module.Prefix))
			return f, true
		}
	}
	return pkgpath, false
}

func main() {
	// either automatically find from repo, or let the user specify the go.mod files
	modules := []Module{
		must(ParseModule("example/go.mod")),
		must(ParseModule("example/hack/json/go.mod")),
	}

	// Sort in reverse prefix length order.
	// When there are multiple prefix matches, the longest prefix is the correct one.
	slices.SortFunc(modules, func(a, b Module) int {
		return -cmp.Compare(len(a.Prefix), len(b.Prefix))
	})

	fmt.Println(TranslatePath("hello/world/main.go", modules))
	fmt.Println(TranslatePath("hello/world/json/main.go", modules))
	fmt.Println(TranslatePath("github.com/external/counter.go", modules))
}

The main idea is to extract module <modulename> from all the go.mod files in the repository, or whatever the user provides. Then whichever longest modulename matches with the package path, use that directory as the root. PS: Using regex (^\s+module\s+(?<modulename>[^ /]+) for extracting module name should work as well, because block comments are not allowed.

If something needs clarification, let me know.

This approach probably won't handle all the corner cases, but it might suffice for 99% cases. And, actually, it might be possible to provide go.mod files together with the coverprofile files -- they should be easily identifiable.

[uhafner]

We prepared a fix in #265. Do you have a small example report that we can use to see if the new parser works better?

[egonelbre]

@uhafner I added a demo into my repo and the coverfile can be found here

When loaded together with all the go.mod files, the path parsing should resolve to these files:

ext/sum.go             => ./hack/ext/sum.go
ext/stat/minmax.go     => ./hack/ext/stat/minmax.go
ext/sub/sub.go         => ./hack/extsub/sub.go
hello.test/main.go     => ./main.go
hello.test/cpu/main.go => ./cpu/main.go