Add function to recursively export all Socrata data to local directory

[tomschenkjr]

The ls.socrata() function supports the listing of all data on a data portal while read.socrata(). Thus, the two can be combined under export.socrata() to download all of the files a neatly compress and place them in a single directory.

This sort of functionality can be used by people wishing to archive data portals or help in the migration of one platform to another one. The function should be focused on saving the data to a local or cloud-based storage (e.g., S3) and should avoid loading all of the data into memory.

Solving #124 will allow an easier integration between ls.socrata() and read.socrata(), so RSocrata v1.7.2-7 or above will be required.

• Output compress CSVs into a directory named after the domain itself (e.g., data.cityofchicago.org/)
• Timestamp each individual file (files should be individually timestamped since an entire portal export may take awhile)
• Ensure that GeoJSON files are exported, compressed (I don't think Add support for geojson downloads #43 needs to be completed first)
• Ensure Shapefiles, KML/KMZ, and other "attach"-based files are also exported, compressed
• Allow users to supply a token in export.socrata() to avoid throttling
• Allow users to choose between CSV or JSON outputs. (CSVs are data.frame friendly, but JSON downloads faster from Socrata)
• Allow users to choose between compression or uncompressed output
• Support output to cloud services such as S3
• Write unit test

An initial alpha is on this gist. This is now on the issue126 branch. Feedback is encouraged as I've done limited testing at this point.

[joshmwolff]

Really glad to see this enhancement request. I've used this function combination before and it works well, so I'm glad to see that you're thinking of combining them. An interesting extension of this enhancement --and one perhaps not vulnerable to throttling -- would be to enable RSocrata users to export a list of column names associated with each dataset (rather than the datasets themselves). That would allow users to investigate which datasets share which fields.

[tomschenkjr]

Thanks, @joshmwolff. It's an interesting idea and can see that being generally useful. But, as I think about it, would that be most useful to retain all of the columns in memory instead of writing them to disk?

[joshmwolff]

@tomschenkjr: think you're right: it would be inefficient to use the export.socrata() function to get a list of all column names for all datasets within a domain if that means downloading all datasets into memory instead of writing them to disk. That said, I'm not sure I'd necessarily want to download all datasets to disk if I were only interested in keeping column names and ultimately combining those names into a single data table. I see, however, that Socrata's Discovery API will in fact return dataset column names for datasets in a particular domain. Can RSocrata hit the Discovery API? That might enable the package to capture the column names without having to also download the column data itself.

[tomschenkjr]

@joshmwolff - right now we're not using the discovery API, but something we're planning for an upcoming release.

In either case, I think read.socrata() could be modified to only extract the column names and not the data. Whether that's through the same SoDA API calls or through the Discovery/Catalog API is probably more a technical/optimization question.

Would you mind opening a new issue on this? I think it's a worthwhile conversation to track as a separate feature than this one.

[tomschenkjr]

Outlining some thoughts on unit tests for this function:

• Count the number of files written to the directory and compare it to the count of data sets listed in ls.socrata().
• Compare the file names written to disk to the names expected to be written (may need to be careful of certain OSes having filename length limitations).
• Ensure the directory created aligns to the domain name being exported.

[joshmwolff]

FWIW, I've been playing with Socrata's "Discovery API" and it works well as a means to for creating a small dataframe of dataset names and other metadata. The following worked for me:

library(jsonlite)
API <- 'http://api.us.socrata.com/api/catalog/v1?only=datasets&domains=data.cambridgema.gov'
RawMetadata <- fromJSON(txt=API)
Metadata <- RawMetadata[['results']]
Metadata <- Metadata$resource

This avoids having to write any files to disk, as you're just storing a single small dataframe in memory (in Cambridge's case, 88 rows by 17 columns).

[tomschenkjr]

Looks interesting and definitely interested in incorporating the Discovery API (i.e., #114).

Would this fit in line with #128?

[joshmwolff]

Whoops. You're right: my comment belongs in #128 rather than here. Feel free to ignore.

[tomschenkjr]

I've pushed the branch to the repo: issue126.

/cc @nicklucius @geneorama

[nicklucius]

I've pushed my first stab at downloading non-tabular data files. Here's how it works: if the first download URL available is not a CSV, the httr::GET() contents are written straight to a file and read.socrata() is not called. The original filename and proper file extension is appended to the filename.

For geographical data with multiple download choices, it looks like KML is first so that is what is being saved.

Still to do:

• compress the file - I had issues with this