Skip to content

About

Web archive for UMD Observatory CCD astronomy data — Python processing pipeline, SQLite + Flask backend, searchable web UI.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

 
 

Latest commit

 

History

46 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Note: This is my fork of warnerem/CCD-data-archive, the UMD Observatory's public data-archive project. My contributions were merged upstream as PRs #1, #2, #4, and #5 (a Google Sheets → SQLite import pipeline, a Flask JSON API, an observation-log parser, and research-page template/UI fixes). — Ivan Wang

The UMD Observatory Data Archive research page, rendering the observation dataset as a searchable table

The live UMD Observatory Data Archive page — the deployed version of this repo's web/7research/templateREAL.html, which my merged PRs #2, #4, and #5 updated.


The original upstream project README follows. The "Prior work" notes and tracking-spreadsheet link are the upstream maintainer's words; the local-setup steps and the Future Works section below are my contribution (merged upstream via PRs #2 and #5, updated here to match the current file layout).

CCD-data-archive

The UMD Observatory has several classes that have collected data on various targets over the years. This project is to make that data available publically.

Prior work was done by Julianna Reese, CPS-SDU student 2020-21 (Elizabeth: emails are in label:work-scholars-2020-21-data_archive and in label:work-observatory-research-data-archive )

SDU Data Archive (https://docs.google.com/spreadsheets/d/1dNfbGlGUQyahnOnTGAttpAUslrAX1JT_BJZCRbYG-nQ/edit?usp=sharing) my spreadsheet where I was keeping track of our datasets, like which ones had been uploaded, etc.

I have updated the files to recreate the UMD Astronomy Open House research page, which now pulls data directly from the updated Google Spreadsheet.

To test this locally, you will need Python 3 with two pip packages: requests and flask.

  1. Run python importSheet.py. This fetches the archive data as CSV and builds a local SQLite database file named data.db. Note that the script pulls from its own working copy of the archive spreadsheet (the URL is hardcoded at the top of the script), which is a separate sheet from the maintainer's tracking spreadsheet linked above.

  2. Run python app.py to start a local Flask server. The JSON API at http://127.0.0.1:5000/api/data serves the rows from data.db.

  3. The Flask / route serves 7research/template.html, a path that no longer exists (the templates were later moved under web/, and no file named template.html remains), so / returns 404 out of the box. To view the Flask-backed page, first copy web/7research/template-w-inc.html to 7research/template.html — that is the template that fetches /api/data — then reload http://127.0.0.1:5000/.

  4. For the CSS and other assets to load, also copy the web folder to a repo-root folder named openhouse (the templates link to their assets under /openhouse/...).

Note that web/7research/templateREAL.html — the page currently in use (see Future Works below) — is the static version: it fetches the Google Sheet directly in the browser via PapaParse and does not use the Flask API. Only template-w-inc.html consumes the Flask API.

There is also a small observation-log parser, txtReader.py. To try it, set filepath at the bottom of the script to example_data/20250624_14in_1998UB17/14in_20250624_log.txt (a sample observing session included in example_data/) and run python txtReader.py.

Please note that this is a work in progress, and aspects of the project may change.

Future Works:

Right now, we are using the HTML in the web/7research file titled templateREAL.html. While this way of doing things is good, it is not ideal. Ideally, having a database backend would be great. Some of the implementation for this could be found in the other template files in the web/7research folder. We would want a backend database as this would both make load times faster as well as make future debugging easier. This is something to look into in the future. The code that I had working with the python files importSheet.py and app.py is made for SQLite and would download the CSV, turn it into a DB, and then turn the DB into a JSON file to be used by the JavaScript. Ideally, if we could get rid of JavaScript from the webpage, that would be great, as some people might not have enabled JavaScript, so the webpage might not function for them without it. Fixing this issue would help with accessibility on the webpage.

About

Web archive for UMD Observatory CCD astronomy data — Python processing pipeline, SQLite + Flask backend, searchable web UI.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages