Docs: Clarify what "issue_id" means

[parker-research]

Clarify -i issue_id vs. the /work/cvdp_<datapoint>/harness/<issue_id>/ directory structure notation

It seems that there are two different things called "issue_id"?

There's the CLI arg -i / --id which seems to be called the "issue_id", and is a value like -i cvdp_agentic_fixed_arbiter_0001 (per the README example).

However, conversely, according to the README_NON_AGENTIC.md document (and likely others), the directory structure ALSO uses an issue_id:

work/cvdp_[datapoint]/
├── harness/[issue_id]/
│   ├── docs/         # Documentation files
│   ├── rtl/          # Hardware description files  
│   ├── verif/        # Verification files
│   ├── rundir/       # Working directory
│   └── src/          # Source files (if applicable)
├── prompts/          # Stored prompts and responses
└── reports/          # Test results and analysis

It seems that the -i argument matches more closely to what's called a "datapoint" into the directory structure docs.

Reconciling this notation, and explaing in what cases the directory structure's issue_id number increments (which is commonly 1) would be very helpful.

I'd propose using "sequence number" to refer to subseqent attempts, in contrast with "ID" which is a reference to an input condition/test case.

I would have thought a new issue_id folder would be created for each k in pass@k (i.e., when k-1 failed, given k>=2), but it seems that's not the case:

./run_samples.py -f example_dataset/cvdp_v1.0.1_example_nonagentic_code_generation_no_commercial_with_solutions.jsonl -l -m gpt-4o-mini -n 5 -k 3 -p work_composite

[npfet]

@parker-research I agree the current terminology is confusing and we need to improve that.

Some background might help: The dataset was created by having common 'project' repositories in which to create questions around. In the example issue-id you gave above, cvdp_agentic_fixed_arbiter_0001, the 'project' is 'cvdp_agentic_fixed_arbiter'. The 0001 then refers to the datapoint '0001' within that project, i.e. the first datapoint that was structured around that project. The datapoint numeric ids are often not consecutive, i.e., there might be a '0001' and '0005'. In that hypothetical case, the IDs 0002, 0003, and 0004 were created but discarded during dataset authoring, or are in the unreleased CVDP problem sets.

It's important to note that each datapoint might have a completely different context. Often times a project 'repo' shares a common code base, but then the context is modified for the individual datapoint. Perhaps the fixed_arbiter is combinational arbitration logic in one datapoint, while another datapoint may have modified it to be sequential. (This is a hypothetical case, I'm not sure if this is true for the fixed_arbiter datapoints currently released.) However, they often share a common original starting point or unit during authoring and branched into individual datapoints from there.

Thus, the individual work/cvdp_[datapoint]/harness/[issue_id]/ directories are referring to the numeric ID of the datapoint within that 'project', even though the individual datapoints are more or less independent of each other.

For the multi-sampling case, there are individual 'work' directories created: e.g., work/sample_1/cvdp_[datapoint]/..., work/sample_2/cvdp_[datapoint]/..., [...], work/sample_5/cvdp_[datapoint]/... in the n=5 case.

Please let me know if this explanation makes sense.