JOSS paper preparation by danielfromearth · Pull Request #1249 · earthaccess-dev/earthaccess

danielfromearth · 2026-03-05T21:51:40Z

Manuscript draft

This PR is intended for revisions and improvements to the manuscript draft being prepared for submission to the Journal of Open Source Software (JOSS).

Paper format: The manuscript is prepared as a Markdown (paper.md) file with references in a paper.bib file, following the JOSS formatting guidelines.

For a PDF preview: With docker installed locally, a PDF preview of the draft manuscript can be generated, by running the following from the earthaccess root directory (as described in the JOSS guidelines's docker section):

docker run --rm \
    --volume $PWD/paper:/data \
    --user $(id -u):$(id -g) \
    --env JOURNAL=joss \
    openjournals/inara

📚 Documentation preview 📚: https://earthaccess--1249.org.readthedocs.build/en/1249/

…JOSS)

github-actions · 2026-03-05T21:51:51Z

👈 Launch a binder notebook on this branch for commit 16fb7b9

I will automatically update this comment whenever this PR is modified

👈 Launch a binder notebook on this branch for commit 38cad6a

👈 Launch a binder notebook on this branch for commit 6af0701

👈 Launch a binder notebook on this branch for commit 767ad52

👈 Launch a binder notebook on this branch for commit dce192c

👈 Launch a binder notebook on this branch for commit ae74db7

👈 Launch a binder notebook on this branch for commit 05f7616

👈 Launch a binder notebook on this branch for commit bb5fd2f

👈 Launch a binder notebook on this branch for commit db3a969

👈 Launch a binder notebook on this branch for commit cf0f975

👈 Launch a binder notebook on this branch for commit 5852fa8

👈 Launch a binder notebook on this branch for commit 1b479c5

👈 Launch a binder notebook on this branch for commit 5029e59

👈 Launch a binder notebook on this branch for commit 2f8cab3

👈 Launch a binder notebook on this branch for commit 0e89b5e

👈 Launch a binder notebook on this branch for commit 5691bf8

👈 Launch a binder notebook on this branch for commit 1af1724

👈 Launch a binder notebook on this branch for commit 81b1384

👈 Launch a binder notebook on this branch for commit 13a9c14

👈 Launch a binder notebook on this branch for commit ce0ada4

👈 Launch a binder notebook on this branch for commit 77428e6

👈 Launch a binder notebook on this branch for commit 06ce30b

👈 Launch a binder notebook on this branch for commit 67feb1f

👈 Launch a binder notebook on this branch for commit 4b25443

👈 Launch a binder notebook on this branch for commit b2d362e

👈 Launch a binder notebook on this branch for commit 4421509

👈 Launch a binder notebook on this branch for commit 86a9b90

👈 Launch a binder notebook on this branch for commit 4d8e3d8

jules32

Hi! Great work on this Danny! A few commits and some suggestions to consider.

Co-authored-by: Julia Stewart Lowndes <julia@openscapes.org>

mfisher87 · 2026-03-05T22:10:04Z

We could symlink this in to our docs!

@mfisher87, want to create an issue for it?

@mfisher87 – pinging you again on this :)

mfisher87 · 2026-03-06T16:06:10Z

after the v1.0.0 release

I would say let's not wait. We've demonstrated impact and I think that matters more.

Alternatively, let's just go 1.0.0 in the short term and be OK with quickly moving to a 2.0.0 release with breaking changes.

I think both are fine, but the latter sets more a precedent of maintainers taking the user impact of breaking changes too lightly.

Co-authored-by: Matt Fisher <3608264+mfisher87@users.noreply.github.com>

danielfromearth · 2026-03-06T21:48:24Z

after the v1.0.0 release

I would say let's not wait. We've demonstrated impact and I think that matters more.

Alternatively, let's just go 1.0.0 in the short term and be OK with quickly moving to a 2.0.0 release with breaking changes.

I think both are fine, but the latter sets more a precedent of maintainers taking the user impact of breaking changes too lightly.

I'm fine with either too. I also think the decision could be on hold until one of the two things – (i) co-author reviews/revisions, (ii) development for v1.0.0 – is completely ready-to-go.

done

danielfromearth · 2026-04-22T14:42:19Z

Friendly ping for co-authors who haven't yet had a chance to review (or at least, approve): @andypbarrett @jhkennedy @jrbourbeau @battistowx @Sherwin-14 @betolink @chuckwondo

Things have been coming together and I think we are getting close to a complete draft that's ready. Would be great to have everyone's eyes on it, even briefly, before we finalize. Could you each take a look in the next week or two?

In particular, please confirm your name, affiliation, and ORCID are correct in the author list. And of course, all other comments welcome.

If timing doesn't work, just comment as such so we know where things stand. Thanks!

Co-authored-by: Jessica Scheick <JessicaS11@users.noreply.github.com>

mfisher87 · 2026-04-23T22:24:06Z

Related: https://earthaccess.zulipchat.com/#narrow/channel/480557-general/topic/JOSS.3F/with/590557057

We're considering / planning going for pyopensci review first, which will give us a stronger review and expedite the JOSS acceptance process if accepted to pyopensci.

Thanks @sampottinger for sharing this with me :)

betolink

I left some comments and suggestions but nothing major, I think this is a good draft so I'm approving as is. Thanks for leading the effort @danielfromearth

betolink · 2026-04-22T18:58:32Z

+
+3. **Access**: Attempts to detect at runtime whether the process is running within AWS `us-west-2`
+   and automatically selects the optimal access path -- direct S3 reads for in-region
+   access or HTTPS downloads otherwise. Users can manually specify an access path if needed.  Files can be opened as `fsspec`-compatible


I like the concise way of presenting this, maybe we can add that being format-agnostic and python file-like object compatible makes the library interoperable with the rest of the scientific python ecosystem (aka Pydata/Pangeo)

betolink · 2026-04-22T19:03:11Z

+open-source tools -- `python-cmr` for search, `fsspec` and `s3fs` for file I/O,
+VirtualiZarr and kerchunk for virtual datasets -- rather than reimplementing their
+functionality. The library's unique contribution is the NASA-specific integration
+layer that binds these tools together.


This is the awesomeness, integrating and simplifying the steps a scientist usually do when working with NASA data. Maybe adding an example of time to science reduction both in lines of code and speed through performance optimizations via fsspec and virtualizarr. Tempo or ICESat-2 can be used for this, before N minutes, now N seconds. Before 10 lines of code, now 1.

Hmm, @betolink, I'm not really sure what would be appropriate for pre-earthaccess code if we want that compared with earthaccess-enabled code... would it be something like this TEMPO S3 Access notebook, which uses boto3, requests, s3fs, etc?

Sorry for the late reply, yes that would be a good comparison, the boto3, requests, CMR response parsing, s3s credentials etc. @danielfromearth

@betolink, I ran some benchmark tests, taking the median of three runs comparing s3.download_file(...) to xr.open_mfdataset(earthaccess.open(results), ...), on the Openscapes 2i2c Jupyterhub. Here is a description of the results that we could add to the paper:

Suggested change

layer that binds these tools together.

layer that binds these tools together. For example, retrieving NASA's TEMPO Level-3

ozone product [@tempo_o3tot_l3] directly from the Earthdata Cloud with `boto3`, `requests`, and `s3fs`

requires about two dozen lines of code (excluding imports) to set up Earthdata Login

authentication, request and periodically refresh temporary S3 credentials, and

serially transfer each granule. The equivalent `earthaccess` workflow is about four

lines (`login()`, `search_data()`, `open()`, and opening the result with `xarray`)

and additionally handles CMR-based discovery. Because

that integration also composes `fsspec` for byte-range access, streaming a single

variable across 28 granules (6.7 GB) and computing a summary statistic completed in

roughly 45 seconds end-to-end, compared with roughly 78 seconds to download the full

files serially beforehand (medians of three runs on the NASA-Openscapes `us-west-2`

JupyterHub). Constructing or reading virtual data stores over the same files with VirtualiZarr or

icechunk, which enabling reading only requested chunks, can reduce access time further.

I would add a corresponding citation for the data too, in the paper.bib.

What do you think of this?

jhkennedy · 2026-04-30T12:48:05Z

Friendly ping for co-authors who haven't yet had a chance to review (or at least, approve): @andypbarrett @jhkennedy @jrbourbeau @battistowx @Sherwin-14 @betolink @chuckwondo

Things have been coming together and I think we are getting close to a complete draft that's ready. Would be great to have everyone's eyes on it, even briefly, before we finalize. Could you each take a look in the next week or two?

In particular, please confirm your name, affiliation, and ORCID are correct in the author list. And of course, all other comments welcome.

If timing doesn't work, just comment as such so we know where things stand. Thanks!

Thanks for the ping @danielfromearth! I've been traveling the last few weeks and will finally make it in back to the office on Monday. I'll have a look ASAP, but I suspect it's already in good shape judging from my quick glance here.

Co-authored-by: Jessica Scheick <JessicaS11@users.noreply.github.com>

jhkennedy

Well, it turns out I do have a bit of feedback 😊 . I think it's in a very good place, and really would be fine to submit with or without my feedback.

Other than the specific things discussed below, I have a pretty big concern around publishing this discussing the automatic cloud-detection logic. That's something we know is technically infeasible to do reliably and we've decided to rip out:
https://github.com/earthaccess-dev/earthaccess/blob/main/docs/governance/decisions/231-aws-us-west-2-checking-method.md

So I'd like to either not mention it or abstract that away in the manuscript language.

Since I have a lot of feedback, I could open a PR into this PR with how I'd resolve my comments, if that's easier. Just let me know.

jhkennedy · 2026-05-05T18:47:48Z

+- **earthdatalogin** [@earthdatalogin_r] provides similar authentication and access
+  functionality for the R programming ecosystem. The two projects share a common motivation and
+  serve as complementary tools for their respective language communities.


🤔 are there other R/Julia things we should call out?

Happy to add others - anyone know?

jhkennedy · 2026-05-05T20:13:53Z

+
+# AI usage disclosure
+
+No generative AI tools were used in the development of the `earthaccess` software; all architectural and design decisions were made exclusively by the authors and contributors.


Hmm, is this true anymore? @betolink have you been using Claude for the virtulizarr work?

I wonder if we need to adopt an AI policy and say something like "...developers may use AI tools but are responsible for their contributions...".

I don't see any problem with modifying this to include development work that leveraged AI. Regarding an AI policy: this question might give another nudge to the AI policy discussion of #1294.

Now this AI policy is being worked on in #1419.

Co-authored-by: Joseph H Kennedy <me@jhkennedy.org>

Co-authored-by: Daniel Kaufman <114174502+danielfromearth@users.noreply.github.com> Co-authored-by: Jessica Scheick <JessicaS11@users.noreply.github.com>

Co-authored-by: Daniel Kaufman <114174502+danielfromearth@users.noreply.github.com>

danielfromearth · 2026-08-19T17:28:47Z

+Several tools exist for accessing NASA Earth science data, each only addressing a specific
+slice of the workflow:
+
+- **python-cmr** [@python_cmr] provides a Python wrapper around the CMR API for dataset


Part of the suggestion from https://github.com/earthaccess-dev/earthaccess/pull/1249/changes#r3190761529:

Suggested change

- **python-cmr** [@python_cmr] provides a Python wrapper around the CMR API for dataset

- **python-cmr** [@python_cmr] handles the metadata step (2) by providing a Python wrapper around the CMR API for dataset

danielfromearth · 2026-08-19T17:29:27Z

+  encapsulate metadata. However, `python-cmr` does not handle authentication, data
+  download, or cloud access -- the areas where researchers face many workflow difficulties.
+
+- **asf_search** [@asf_search] sits between `python-cmr` and `earthaccess`: it focuses on search and discovery but also handles authentication and access. It emerged around the same time as `earthaccess`, from similar access challenges. Unlike `earthaccess`, which is mission- and domain-agnostic, `asf_search` is tailored to synthetic aperture radar (SAR) data, adding domain-specific tooling and functionality.


Part of the suggestion from https://github.com/earthaccess-dev/earthaccess/pull/1249/changes#r3190761529:

Suggested change

- **asf_search** [@asf_search] sits between `python-cmr` and `earthaccess`: it focuses on search and discovery but also handles authentication and access. It emerged around the same time as `earthaccess`, from similar access challenges. Unlike `earthaccess`, which is mission- and domain-agnostic, `asf_search` is tailored to synthetic aperture radar (SAR) data, adding domain-specific tooling and functionality.

- **asf_search** [@asf_search] sits between `python-cmr` and `earthaccess`: it focuses on search and discovery but also handles authentication and access (workflow steps 1–5). It emerged around the same time as `earthaccess`, from similar access challenges. Unlike `earthaccess`, which is mission- and domain-agnostic, `asf_search` is tailored to synthetic aperture radar (SAR) data, adding domain-specific tooling and functionality.

danielfromearth · 2026-08-19T17:30:05Z

+- **asf_search** [@asf_search] sits between `python-cmr` and `earthaccess`: it focuses on search and discovery but also handles authentication and access. It emerged around the same time as `earthaccess`, from similar access challenges. Unlike `earthaccess`, which is mission- and domain-agnostic, `asf_search` is tailored to synthetic aperture radar (SAR) data, adding domain-specific tooling and functionality.
+
+
+- **icepyx** [@icepyx] provides a mission-specific solution to the general problem solved by earthaccess: search, discovery, authentication, and access to ICESat-2 [@icesat2] data products. Established about a year before earthaccess, it now leverages earthaccess via a mixin to provide authentication and tokening when users perform an action that requires logging in.


Part of the suggestion from https://github.com/earthaccess-dev/earthaccess/pull/1249/changes#r3190761529:

Suggested change

- **icepyx** [@icepyx] provides a mission-specific solution to the general problem solved by earthaccess: search, discovery, authentication, and access to ICESat-2 [@icesat2] data products. Established about a year before earthaccess, it now leverages earthaccess via a mixin to provide authentication and tokening when users perform an action that requires logging in.

- **icepyx** [@icepyx] provides a mission-specific solution to the general problem solved by earthaccess: search, discovery, authentication, and access to ICESat-2 [@icesat2] data products. Established about a year before earthaccess, it handles workflow steps 1, 2, and 5 on its own and now leverages earthaccess via a mixin to provide authentication and tokening when users perform an action that requires logging in.

danielfromearth · 2026-08-19T17:31:06Z

+- **earthdatalogin** [@earthdatalogin_r] provides similar authentication and access
+  functionality for the R programming ecosystem. The two projects share a common motivation and


Part of the suggestion from https://github.com/earthaccess-dev/earthaccess/pull/1249/changes#r3190761529:

Suggested change

- **earthdatalogin** [@earthdatalogin_r] provides similar authentication and access

functionality for the R programming ecosystem. The two projects share a common motivation and

- **earthdatalogin** [@earthdatalogin_r] provides similar authentication and access

functionality (steps 3–5) for the R programming ecosystem. The two projects share a common motivation and

danielfromearth added 2 commits March 3, 2026 14:52

add initial manuscript drafting for Journal of Open Source Software (…

e709155

…JOSS)

add content and revise manuscript draft

16fb7b9

jules32 approved these changes Mar 5, 2026

View reviewed changes

Comment thread paper/paper.md Outdated

Comment thread paper/paper.md

Comment thread paper/paper.md Outdated

Comment thread paper/paper.md Outdated

Comment thread paper/paper.md

Comment thread paper/paper.md Outdated

danielfromearth and others added 4 commits March 6, 2026 09:48

Apply suggestions from code review

38cad6a

Co-authored-by: Julia Stewart Lowndes <julia@openscapes.org>

Apply revisions from @jules32 review comments

6af0701

add icepack example

767ad52

add blurb about contributing upstream

dce192c

mfisher87 previously requested changes Mar 6, 2026

View reviewed changes

Merge branch 'main' into joss-paper

ae74db7

danielfromearth mentioned this pull request Mar 6, 2026

Add entries to citation file #1254

Merged

Update affiliations per code review

05f7616

Co-authored-by: Matt Fisher <3608264+mfisher87@users.noreply.github.com>

danielfromearth changed the title ~~Joss paper~~ JOSS paper preparation Mar 6, 2026

add entries to authors and affiliations

bb5fd2f

danielfromearth requested a review from andypbarrett March 10, 2026 19:07

danielfromearth assigned betolink, itcarroll, chuckwondo, jhkennedy, jrbourbeau, JessicaS11, battistowx, asteiker and Sherwin-14 and unassigned betolink, itcarroll and chuckwondo Mar 10, 2026

Update contributors listing per @JessicaS11 comment

13a9c14

Co-authored-by: Jessica Scheick <JessicaS11@users.noreply.github.com>

danielfromearth dismissed JessicaS11’s stale review via 13a9c14 April 22, 2026 14:43

itcarroll previously approved these changes Apr 22, 2026

View reviewed changes

Merge branch 'main' into joss-paper

ce0ada4

asteiker mentioned this pull request Apr 22, 2026

Consider creating and adding an "output register" to Readthedocs #1216

Open

Sherwin-14 approved these changes Apr 23, 2026

View reviewed changes

Merge branch 'main' into joss-paper

77428e6

betolink previously approved these changes Apr 29, 2026

View reviewed changes

Apply suggestions from code review

06ce30b

Co-authored-by: Jessica Scheick <JessicaS11@users.noreply.github.com>

danielfromearth dismissed stale reviews from betolink and itcarroll via 06ce30b May 1, 2026 18:10

jhkennedy reviewed May 5, 2026

View reviewed changes

danielfromearth and others added 4 commits June 19, 2026 13:53

Updates per code review: an affiliation, HTTP access, and community

67feb1f

Co-authored-by: Joseph H Kennedy <me@jhkennedy.org>

Add references for earthaccess-dependents

4b25443

add bibtex entries for earthaccess-dependent references

b2d362e

Add asf_search entry

4421509

JessicaS11 reviewed Jul 21, 2026

View reviewed changes

Comment thread paper/paper.bib

danielfromearth mentioned this pull request Aug 14, 2026

earthaccess: Python Library for NASA Earthdata APIs pyOpenSci/software-submission#302

Open

31 tasks

danielfromearth commented Aug 17, 2026

View reviewed changes

Comment thread paper/paper.md Outdated

Add icesat2 ref, and remove harmony-py - per code review

86a9b90

Co-authored-by: Daniel Kaufman <114174502+danielfromearth@users.noreply.github.com> Co-authored-by: Jessica Scheick <JessicaS11@users.noreply.github.com>

danielfromearth commented Aug 19, 2026

View reviewed changes

Comment thread paper/paper.md Outdated

Update data access workflow description and steps per code review

4d8e3d8

Co-authored-by: Daniel Kaufman <114174502+danielfromearth@users.noreply.github.com>

danielfromearth commented Aug 19, 2026

View reviewed changes

danielfromearth added the impact: documentation Use this label for issues related to Improvements or additions to documentation. label Aug 28, 2026

-layer that binds these tools together.
+layer that binds these tools together. For example, retrieving NASA's TEMPO Level-3
+ozone product [@tempo_o3tot_l3] directly from the Earthdata Cloud with `boto3`, `requests`, and `s3fs`
+requires about two dozen lines of code (excluding imports) to set up Earthdata Login
+authentication, request and periodically refresh temporary S3 credentials, and
+serially transfer each granule. The equivalent `earthaccess` workflow is about four
+lines (`login()`, `search_data()`, `open()`, and opening the result with `xarray`)
+and additionally handles CMR-based discovery. Because
+that integration also composes `fsspec` for byte-range access, streaming a single
+variable across 28 granules (6.7 GB) and computing a summary statistic completed in
+roughly 45 seconds end-to-end, compared with roughly 78 seconds to download the full
+files serially beforehand (medians of three runs on the NASA-Openscapes `us-west-2`
+JupyterHub). Constructing or reading virtual data stores over the same files with VirtualiZarr or
+icechunk, which enabling reading only requested chunks, can reduce access time further.


		# AI usage disclosure

		No generative AI tools were used in the development of the `earthaccess` software; all architectural and design decisions were made exclusively by the authors and contributors.

	- python-cmr [@python_cmr] provides a Python wrapper around the CMR API for dataset
	- python-cmr [@python_cmr] handles the metadata step (2) by providing a Python wrapper around the CMR API for dataset

		- asf_search [@asf_search] sits between `python-cmr` and `earthaccess`: it focuses on search and discovery but also handles authentication and access. It emerged around the same time as `earthaccess`, from similar access challenges. Unlike `earthaccess`, which is mission- and domain-agnostic, `asf_search` is tailored to synthetic aperture radar (SAR) data, adding domain-specific tooling and functionality.


		- icepyx [@icepyx] provides a mission-specific solution to the general problem solved by earthaccess: search, discovery, authentication, and access to ICESat-2 [@icesat2] data products. Established about a year before earthaccess, it now leverages earthaccess via a mixin to provide authentication and tokening when users perform an action that requires logging in.

		- earthdatalogin [@earthdatalogin_r] provides similar authentication and access
		functionality for the R programming ecosystem. The two projects share a common motivation and

Uh oh!

Conversation

danielfromearth commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Manuscript draft

Uh oh!

github-actions Bot commented Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jules32 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mfisher87 commented Mar 6, 2026

Uh oh!

danielfromearth commented Mar 6, 2026

Uh oh!

danielfromearth commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mfisher87 commented Apr 23, 2026

Uh oh!

betolink left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danielfromearth Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jhkennedy commented Apr 30, 2026

Uh oh!

jhkennedy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danielfromearth Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

danielfromearth commented Mar 5, 2026 •

edited

Loading

github-actions Bot commented Mar 5, 2026 •

edited

Loading

danielfromearth commented Apr 22, 2026 •

edited

Loading

danielfromearth Aug 19, 2026 •

edited

Loading

danielfromearth Jul 9, 2026 •

edited

Loading

danielfromearth Aug 19, 2026 •

edited

Loading