Note: This is my fork of warnerem/CCD-data-archive, the UMD Observatory's public data-archive project. My contributions were merged upstream as PRs #1, #2, #4, and #5 (a Google Sheets → SQLite import pipeline, a Flask JSON API, an observation-log parser, and research-page template/UI fixes). — Ivan Wang
The live UMD Observatory Data Archive page — the deployed version of this repo's web/7research/templateREAL.html, which my merged PRs #2, #4, and #5 updated.
The original upstream project README follows. The "Prior work" notes and tracking-spreadsheet link are the upstream maintainer's words; the local-setup steps and the Future Works section below are my contribution (merged upstream via PRs #2 and #5, updated here to match the current file layout).
The UMD Observatory has several classes that have collected data on various targets over the years. This project is to make that data available publically.
Prior work was done by Julianna Reese, CPS-SDU student 2020-21 (Elizabeth: emails are in label:work-scholars-2020-21-data_archive and in label:work-observatory-research-data-archive )
SDU Data Archive (https://docs.google.com/spreadsheets/d/1dNfbGlGUQyahnOnTGAttpAUslrAX1JT_BJZCRbYG-nQ/edit?usp=sharing) my spreadsheet where I was keeping track of our datasets, like which ones had been uploaded, etc.
I have updated the files to recreate the UMD Astronomy Open House research page, which now pulls data directly from the updated Google Spreadsheet.
To test this locally, you will need Python 3 with two pip packages: requests and flask.
-
Run
python importSheet.py. This fetches the archive data as CSV and builds a local SQLite database file nameddata.db. Note that the script pulls from its own working copy of the archive spreadsheet (the URL is hardcoded at the top of the script), which is a separate sheet from the maintainer's tracking spreadsheet linked above. -
Run
python app.pyto start a local Flask server. The JSON API athttp://127.0.0.1:5000/api/dataserves the rows fromdata.db. -
The Flask
/route serves7research/template.html, a path that no longer exists (the templates were later moved underweb/, and no file namedtemplate.htmlremains), so/returns 404 out of the box. To view the Flask-backed page, first copyweb/7research/template-w-inc.htmlto7research/template.html— that is the template that fetches/api/data— then reloadhttp://127.0.0.1:5000/. -
For the CSS and other assets to load, also copy the
webfolder to a repo-root folder namedopenhouse(the templates link to their assets under/openhouse/...).
Note that web/7research/templateREAL.html — the page currently in use (see Future Works below) — is the static version: it fetches the Google Sheet directly in the browser via PapaParse and does not use the Flask API. Only template-w-inc.html consumes the Flask API.
There is also a small observation-log parser, txtReader.py. To try it, set filepath at the bottom of the script to example_data/20250624_14in_1998UB17/14in_20250624_log.txt (a sample observing session included in example_data/) and run python txtReader.py.
Please note that this is a work in progress, and aspects of the project may change.
Future Works:
Right now, we are using the HTML in the web/7research file titled templateREAL.html. While this way of doing things is good, it is not ideal. Ideally, having a database backend would be great. Some of the implementation for this could be found in the other template files in the web/7research folder. We would want a backend database as this would both make load times faster as well as make future debugging easier. This is something to look into in the future. The code that I had working with the python files importSheet.py and app.py is made for SQLite and would download the CSV, turn it into a DB, and then turn the DB into a JSON file to be used by the JavaScript. Ideally, if we could get rid of JavaScript from the webpage, that would be great, as some people might not have enabled JavaScript, so the webpage might not function for them without it. Fixing this issue would help with accessibility on the webpage.
