WIP: 109 layer descriptor shape parameters

[brian-r-calder]

Issue #109: layers do not currently have separate shape parameters (except Geometadata layer), but should in order for VR refinements to work appropriately. This pull request adds this functionality, and also bug-fixes a number of issues in the VR implementation, and other deprecation issues (e.g., snprintf() rather than sprintf()).

[selimnairb]

@brian-r-calder This branch is being taken when running test_bag_vrmetadata.cpp, specifically when the test attempts to read back the VR metadata. Is the test wrong, or is there a mistake in Layer::read()?

[brian-r-calder]

I don't see a branch here, but the code is correct --- we have to read the dimensions specifically from the layer rather than from the dataset, since we need to allow each layer to have its own dimensions.

[selimnairb]

What I meant was if (columnEnd >= numColumns || rowEnd >= numRows) evaluates to true when Layer::read() is called from test_bag_vrmetadata.cpp:452. Is the VR layer being created by the test incorrect?

[selimnairb]

@selimnairb Make sure this PR addresses #115.

[selimnairb]

This PR should be merged after #143

[selimnairb]

varres_refinements were created in BAG 1.6.3 using the following code:

HD5 type creation

bag_opt_group.c::bagCreateVarResRefinementGroup(): 101-120:

bagError bagCreateVarResRefinementGroup(bagHandle hnd, bagData *data, u32 const n_cells)
{
 bagError err;
 herr_t  herr;
 
 if ((data->opt[VarRes_Refinement_Group].datatype = H5Tcreate(H5T_COMPOUND, sizeof(bagVarResRefinementGroup))) < 0) {
  return BAG_HDF_TYPE_CREATE_FAILURE;
 }
 herr = H5Tinsert(data->opt[VarRes_Refinement_Group].datatype, "depth", HOFFSET(bagVarResRefinementGroup, depth), H5T_NATIVE_FLOAT);
 if (herr < 0) return BAG_HDF_TYPE_CREATE_FAILURE;
 herr = H5Tinsert(data->opt[VarRes_Refinement_Group].datatype, "depth_uncrt", HOFFSET(bagVarResRefinementGroup, depth_uncrt), H5T_NATIVE_FLOAT);
 if (herr < 0) return BAG_HDF_TYPE_CREATE_FAILURE;
 
 data->opt[VarRes_Refinement_Group].nrows = 1;
 data->opt[VarRes_Refinement_Group].ncols = n_cells;
 
 err = bagCreateOptionalDataset(hnd, data, VarRes_Refinement_Group);
 
 return err;
}

bagCreateVarResRefinementGroup() calls bagCreateOptionalDataset() (see below), so bagCreateVarResRefinementGroup() is the entrypoint to creating varres_refinements. bagCreateVarResRefinementGroup() is in turn called by bagCreateVariableResolutionLayers(). So bagCreateVariableResolutionLayers() is what BAG library users use to create VR bags. See, for example, examples/bag_vr_create.c:177.

bagCreateVarResRefinementGroup()sets opt[VarRes_Refinement_Group].nrows = 1.

HD5 group creation

bag_opt_surfaces.c::bagCreateOptionalDataset():352-355:

if ((dataset_id = H5Dcreate(file_id, VARRES_REFINEMENT_GROUP_PATH, datatype_id, dataspace_id, plist_id)) < 0) {
    status = H5Fclose(file_id);
    return BAG_HDF_CREATE_GROUP_FAILURE;
}

The dataspace is defined at bag_opt_surfaces.c::bagCreateOptionalDataset():190-194:

if ((dataspace_id = H5Screate_simple(RANK, dims, NULL)) < 0)
{
    status = H5Fclose (file_id);
    return (BAG_HDF_CREATE_DATASPACE_FAILURE);
}

dims is defined at bag_opt_services.c:: bagCreateOptionalDataset():128:

hsize_t      dims[RANK];

RANK is defined at bag_private.h:48:

#define RANK 2

And dims dimensions are initialized at bag_opt_services.c:: bagCreateOptionalDataset():158-159:

dims[0] = data->opt[type].nrows;
dims[1] = data->opt[type].ncols;

Conclusions on BAG 1.6.3 handling of varres_refinements

Since RANK is always 2, BAG 1.6.3 is creating a dataspace for varres_refinements that is of shape [1, n_cells], where n_cellsis passed in to bagCreateVarResRefinementGroup(). Thus, this is a 2D array, treated as a 1D array.

Changes required for BAG 2.0.x to have the same behavior as BAG 1.6.3

What changes are needed to ensure that varres_refinements are of shape [1, n_cells] in BAG 2.0.x? Perhaps none. This may already be fixed in branches 109-LayerDescriptor-shape-parameters and 109-LayerDescriptor-shape-parameters-update-testcases (the latter being a child of the former; the child improving test cases).

To wit, in branch 109-LayerDescriptor-shape-parameters-update-testcases in test_bag_vrrefinements.cpp:461-496:

 UNSCOPED_INFO("Write another VR refinement.");
{
    constexpr BAG::VRRefinementsItem kExpectedItem0{10.8f, 1.654f};

    const auto* buffer = reinterpret_cast<const uint8_t*>(&kExpectedItem0);

    constexpr uint32_t kRowStart = 0;  // unused
    constexpr uint32_t kColumnStart = 1;
    constexpr uint32_t kRowEnd = 0;  // unused
    constexpr uint32_t kColumnEnd = 1;

    REQUIRE_NOTHROW(pVrRefinements->write(kRowStart, kColumnStart, kRowEnd,
        kColumnEnd, buffer));

    UNSCOPED_INFO("Read the record back.");
    auto result = pVrRefinements->read(kRowStart, kColumnStart, kRowEnd, kColumnEnd);
    CHECK(result);

    const auto* res = reinterpret_cast<const BAG::VRRefinementsItem*>(result.data());
    UNSCOPED_INFO("Check the expected value of VRRefinementItem::depth.");
    CHECK(res->depth == kExpectedItem0.depth);
    UNSCOPED_INFO("Check the expected value of VRRefinementItem::depth_uncrt.");
    CHECK(res->depth_uncrt == kExpectedItem0.depth_uncrt);
}

// Re-open BAG file readonly and verify varres_refinements layer dimensions...
pDataset->close();
const auto dataset = Dataset::open(tmpBagFile, BAG_OPEN_READONLY);
REQUIRE(dataset);
auto vrRef = dataset->getVRRefinements();
REQUIRE(vrRef);
const auto vrRefDesc = vrRef->getDescriptor();
auto vrRefDescDims = vrRefDesc->getDims();
CHECK(std::get<0>(vrRefDescDims) == 1);
CHECK(std::get<1>(vrRefDescDims) == 2);
}

We create a BAG with two VR refinements, and see the varres_refinements dataspace shape is [1, 2], so a 2D array with a single row and two columns.

Since we are also able to read VR BAGs written with BAG 1.6.3, we should be okay? For example, see test_vr_bag.cpp:87-124:

TEST_CASE("test VR BAG reading libbag 1.6.3", "[dataset][open][VR][1.6.3]")
{
    const std::string bagFileName{std::string{std::getenv("BAG_SAMPLES_PATH")} +
                                  "/bag_163_vr.bag"};

    const size_t kNumExpectedLayers = 4;  // Elevation, Uncertainty, varres_metadata, varres_refinements

    SECTION("open read only")
    {
        const auto dataset = Dataset::open(bagFileName, BAG_OPEN_READONLY);
        REQUIRE(dataset);

        CHECK(dataset->getLayerTypes().size() == kNumExpectedLayers);

        const uint32_t kExpectedRows = 529;
        const uint32_t kExpectedCols = 579;
        CHECK(dataset->getDescriptor().getVersion() == "1.6.3");
        auto dims = dataset->getDescriptor().getDims();
        CHECK(std::get<0>(dims) == kExpectedRows);
        CHECK(std::get<1>(dims) == kExpectedCols);

        auto vrMeta = dataset->getVRMetadata();
        REQUIRE(vrMeta);
        const auto vrMetaDesc = vrMeta->getDescriptor();
        auto vrMetaDescDims = vrMetaDesc->getDims();
        // VR metadata descriptor dims should be the same as BAG dataset dims...
        CHECK(std::get<0>(vrMetaDescDims) == kExpectedRows);
        CHECK(std::get<1>(vrMetaDescDims) == kExpectedCols);

        // Verify varres_refinements layer dimensions...
        auto vrRef = dataset->getVRRefinements();
        REQUIRE(vrRef);
        const auto vrRefDesc = vrRef->getDescriptor();
        auto vrRefDescDims = vrRefDesc->getDims();
        CHECK(std::get<0>(vrRefDescDims) == 1);
        CHECK(std::get<1>(vrRefDescDims) == 2957372);
    }
}

Unless there are other VR BAGs in existance that truely use a 1D array to store VR refinements, we shouldn't need any further changes to the BAG 2.0.x code. In this case, we should update the FSD to clarify the format of the varres_refinements layer. The current language is:

The refinement layer provides the depth and uncertainty estimates for the refined grids, and is stored at /BAG_Root/varres_refinements, which consists of the row-major concatenation HDF5-fields described in Table E1.4.

This should probably be updated to something like:

The refinement layer provides the depth and uncertainty estimates for the refined grids, and is stored at /BAG_Root/varres_refinements, which consists of the row-major concatenation HDF5-fields described in Table E1.4. The `varres_refinements` layer must be a 2D array whose dataspace is of shape [1, N] where N is the number of refinements.

This all said, how does BAG 2.0.x (109-LayerDescriptor-shape-parameters) create varres_refinements?

VRRefinements::createH5dataset():bag_vrrefinements.cpp:215-216:

const auto h5dataSet = h5file.createDataSet(VR_REFINEMENT_PATH,
    memDataType, h5fileDataSpace, h5createPropList);

And the h5fileDataSpace is defined at VRRefinements::createH5dataset():bag_vrrefinements.cpp:185-187:

std::array<hsize_t, kRank> fileDims{0, 0};
const std::array<hsize_t, kRank> kMaxFileDims{H5S_UNLIMITED, H5S_UNLIMITED};
const ::H5::DataSpace h5fileDataSpace{kRank, fileDims.data(), kMaxFileDims.data()};

where kRank is defined at bag_types.h:17-18:

// The rank of simple layers.
constexpr static int kRank = 2;

Then the varres_refinements layer is updated (i.e., expanded as needed and then written to) VRRefinements::writeProxy():bag_vrrefinements.cpp:328-334, expansion:

if ((fileDims[0] < (rowEnd + 1)) ||
    (fileDims[1] < (columnEnd + 1)))
{
    const std::array<hsize_t, kRank> newDims{
            std::max<hsize_t>(fileDims[0], rowEnd + 1),
            std::max<hsize_t>(fileDims[1], columnEnd + 1)};
    m_pH5dataSet->extend(newDims.data());
...

The the write happens at VRRefinements::writeProxy():bag_vrrefinements.cpp:357:

m_pH5dataSet->write(buffer, memDataType, memDataSpace, fileDataSpace);

[selimnairb]

@selimnairb Make sure this PR addresses #115.

Discussed with @anthonyklemm, will address #115 after #109 is merged.