RSLC missing data mask - #370
Conversation
Document get_valid_pulse_mask, find_valid_pulse_intervals, split_segments, and join_segments to clarify their inputs/outputs for the missing-pulse valid data mask logic. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
This work probably supersedes #344 |
|
Thank you @bhawkins for the PR . The listed items reflects what we discussed offline. Once the masks are more mature we will derive couple of scalar parameters representing missing data in the observation used for focusing the image. |
…rsection The row-by-row scanline intersection against every polygon was dominated by per-call Shapely/GEOS overhead. Build one in-memory OGR layer from the polygons and rasterize each output block with gdal.RasterizeLayer instead, which also correctly handles arbitrary (non-quadrilateral) polygon shapes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Split get_focused_sub_swaths into get_focused_sub_swath_polygons (bbox to polygon conversion + reskew) and the existing rasterize-to-indices step, so the polygon computation can be reused. Add save_valid_data_mask, which pairs that polygon computation with save_subswath_polygons_to_image to write a full-resolution valid-data mask directly instead of per-pulse indices. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bring save_valid_data_mask in line with the bit/num_valid support added to save_subswath_polygons_to_image.
Extract raw sub-swath bbox computation (per-observation lookup, chirp duration, seamless-observation merging) into get_raw_sub_swath_bboxes so it can be shared between get_focused_sub_swaths and a new save_valid_data_mask, mirroring the isce3-level refactor of valid_regions.py.
Replace the bit argument with a lookup keyed on out_chan.pol so each polarization channel gets its own fixed bit, and add PolValidMask (public IntFlag) so callers can decode which channel(s) mark a pixel valid.
getSubSwathBboxes was relying on its default polarization (first available for the frequency), not necessarily the channel actually being processed. Pass polarization explicitly, and have save_valid_data_mask request use_rx_pulse_mask=True so the per-pol valid-data mask accounts for pulseHasValidSamples metadata specific to the receive channel.
|
Alice implemented the necessary L0B changes. That wound up revealing that HV is mostly valid for this test image, making it an even better test than I thought. I implemented the RSLC features, though there are a couple subtleties. One is that the validSamplesSubSwath mask now excludes regions that are invalid on all polarizations. That turned out to be easier than trying guess about missing information (e.g., handling the case where data is missing at a DWP update). I assume that's unobjectionable. The other is that I implemented a single mask layer, assigning one bit to each of the possible polarizations. I figured that's a bit more efficient than having a separate mask image for each pol. I just called the dataset "mask" which is consistent with GSLC and GCOV products, at least. Open to suggestions. If we weren't concerned about backwards compatibility we could combine this new mask with the inputDataExceptionMask and also delete the validSamplesSubSwathX datasets. 
|
|
Oh yeah, I uploaded this RSLC product to s3://nisar-adt/bhawkins/scratch/rslc_mask_example.h5 if anyone wants a sample product to play with. |
|
Following this morning's discussion, I modified the code to change inputDataExceptionMask from uint8 to uint16 and use the new bits to store the valid data mask. This reduces the number of mask layers users have to read and minimizes changes to downstream workflows. I updated the description to read
I posted a sample data product to s3://nisar-adt/bhawkins/scratch/rslc_mask_example_20260831.h5 |
|
Thanks @bhawkins for updating this PR based on few offline discussions.
HH: ~31% valid
We know that for this product HV is fine and HH has missing data. Therefore, I think the second approach is more accurate to demonstrate the percentage of valid data in term of missing/corrupted data. The impact will be even more pronounced when we have fixed PRF with multiple transmit gaps. What do you think? Is it possible to add this metric? |
Tyler-g-hudson
left a comment
Tyler-g-hudson
left a comment
There was a problem hiding this comment.
Some quick review
| Total number of valid pixels in the image. | ||
| """ | ||
| bit = int(_PolBit[out_chan.pol]) | ||
|
|
There was a problem hiding this comment.
Can calculate min_segment_length based on azres here.
There was a problem hiding this comment.
Do you think maybe this minimum segment length concept could be inverted? That is to say, would it be better to have a "maximum missing rangelines per aperture" with the additional stipulation that those rangelines can't be adjacent?
There was a problem hiding this comment.
I don't think that would sufficiently constrain the distribution of missing pulses. For example, if all the missing pulses were clustered in one spot but interleaved with one pulse of valid data, the effect on image quality is a lot worse than if they were distributed evenly throughout the aperture. Requiring contiguous blocks of valid data provides a stronger guarantee that regions marked as "valid" are actually uncorrupted.
|
@xhuang-jpl this is the PR that I brought up to you offline just now |
…ngth Mirrors the change already made to save_valid_data_mask: derive min_segment_length from min_segment_fraction and the synthetic aperture length instead of taking a fixed pulse count.
|
Okay, I think I've addressed all the comments from yesterday's review. The main changes were
@hfattahi I guess you weren't kidding about not seeing attributes 😆 . The mask dataset already has attributes that calculate what you described as option 1. Note that the sub-swath vectors will have null areas wherever both channels have missing data, so with option 2 one of the polarizations would have 100% valid by construction, even if tons of data were missing. I implemented the option I mentioned a week or two ago where the metric is the ratio of valid pulses to total pulses in the subset of the L0B that is used to create the RSLC. Let me know what you think: 
|
Haha yeah I was not joking! Thanks @bhawkins I think the metrics are great and representative. For each of those we would give the min/max of the attributes you have added for all polarizations. Therefore, regardless of how many frequency or polarizations a product may have, the identification will always have these four datasets. what do you think? |
3 participants
This is work in progress for improving the RSLC valid data mask. Currently, the RSLC mask only considers the transmit gaps and the range location of the swath (which may change every 10 s). We want it to also reflect missing data, typically echoes that were recorded on board but were lost in the downlink. Here's the contents of
validSamplesSubSwath1for a missing data case using this branch as of today (green is start, red is stop):Based on today's discussion the code needs substantial changes, but I thought it'd be helpful to open PR so it could be added to the delivery milestone, if nothing else. The core problem is that the existing metadata is not polarization-specific, so it can't represent polarization-specific missing data masks. Here are the changes we discussed:
validSamplesSubSwath1. If the gap metadata is missing for one RX polarization but not the other, then it should be populated with the valid one.pulseHasValidSamplesthat report missing pulses. That saves users the trouble of reading the whole image and checking whether all the samples are fill_value in each row.validSamplesSubSwath1should basically just reflect the TX gaps and range timing as it currently does.