Find a way to search data packages more reliably

[roll]

Overview

Currently, it's not reproducible. Every new Github Search will return a different set of packages. It would be not critical it it wouldn't miss some popular ones like https://github.com/datasets/covid-19

We can start doing commutative searches - appending new repositories found to data/packages.csv instead of re-creating it every time from a scratch.

Also, we might consider a manual way to add data package e.g. from a curated list.

---

Probably we can use Frictionless Transform just merging data/pakages.raw.csv and data/packages.csv

[RandomFractals]

hey @roll that packages.csv looks great.

I am considering using that CSV doc for populating new built-in Data Packages gallery in Tabular Data viewer vscode extension via new tabular.data.listDataPackages command and quick file picker:

RandomFractals/tabular-data-viewer#102

I think it will work nicely as long as we use your packages.csv in this repo as the master for our datapackage.json listings.

Packages.csv in Perspective Grid view to see what's listed there:

Minor request: would it be possible to omit content column in the compiled packages.csv?

Otherwise, that file is too large to load as a master list dynamically. Looks like the url to the corresponding datapackage.json for each dataset can be easily constructed from other data columns and loaded separately.

With that, until you have your cron job with github actions update that list daily, I'll probably just use a trimmed down version of it for the new Data Packages feature I plan to release in Tabular Data Viewer next.

[RandomFractals]

Here is a pic of the upcoming Data Packages support with that list in vscode:

[roll]

Great work @RandomFractals!

This project needs this package contents but you can drop it in your pipeline before packing it into the extension (or you can cherry-pick the data you need from there)

[RandomFractals]

@roll yeah, I just used trimmed down version of your compiled list for now.

Selecting a data package now lists tabular data resources from it in a quick pick list and allows you to load and view that tabular dataset in vscode. I'll be shipping that feature soon: RandomFractals/tabular-data-viewer#104

Hopefully some of the data package format adopters using vscode IDE will find it useful for quick EDAs.