Are we providing a coherent user experience regarding search and open?

[itcarroll]

Transferring to discussion a couple conversations (one taken up in #770, which I'm closing in favor of this Discussion, and one from 2024-01-07 Hackday) regarding top-level earthaccess user experience, particularly search* and open* functions.

We describe our public API within earthaccess.__init__.py as follows.

earthaccess/earthaccess/__init__.py

Lines 33 to 67 in 91866ac

	__all__ = [
	# api.py
	"login",
	"search_datasets",
	"search_data",
	"search_services",
	"get_requests_https_session",
	"get_fsspec_https_session",
	"get_s3fs_session",
	"get_s3_credentials",
	"get_s3_filesystem",
	"get_edl_token",
	"granule_query",
	"collection_query",
	"open",
	"download",
	"auth_environ",
	# search.py
	"DataGranule",
	"DataGranules",
	"DataCollection",
	"DataCollections",
	"DataServices",
	# auth.py
	"Auth",
	# store.py
	"Store",
	# kerchunk
	"consolidate_metadata",
	# virtualizarr
	"open_virtual_dataset",
	"open_virtual_mfdataset",
	"PROD",
	"UAT",
	]

My summary of the current status:

1. We define three search* entry points in the api module: search_datasets, search_data, and search_services. Discussion in consider alternative names for search_data and search_datasets #770 was tending toward a renaming of search_ to find_ (thanks @chuckwondo for the excellent analogy to search_bags), and also towards renaming _datasets to _collections and _data to _items (although there was not consensus on _items) to align with STAC vocabulary.
2. We provide granule_query and collection_query as "lower-level" (i.e. more python-cmr like) access to CMR results.
3. We define one open entry point in the api module and two open_virtual_* entry points in the dmrpp_zarr module.
   • Our open returns a list of file-like objects like you get from with fsspec.open_files(...) as.
   • Our open_virtual_[mf]dataset returns a dataset like you get from xarray.open_[mf]dataset.

Take this thread anywhere you want, but the concern I have is whether each new underscore detracts from the user experience. I'd like to think a user should not even need to be familiar with the difference between CMR collection vs granule queries nor have to know that the data accessed comes from a "virtual" dataset.

The work being done in #898 and towards smart_open should be kept in mind here too.

[itcarroll]

I've been kicking the tires on an idea in my head, so will give others the chance too as well.

Could a single search method that returns a "results" object (i.e. object that behaves like a list of DataCollections) be the only CMR entry point? The "results" object needs a filter method that applies granule query parameters and returns a different "results" object that behaves like a list of DataGranules but also like an fsspec.filesystem. It gives you ls as well as the low-level open. You can also pass this "results" object to smart_open (or, might I suggest access) to return a suitable xarray object.

• A "search" followed by a "filter" matches the EarthData Search flow, where you find a collection, then narrow down the granules you want from a collection, then add it to your "order".
• Using the word "search" here with no suffix avoids the search_bags concern.
• I don't know enough about search_services to say whether that could be worked in too.
• Two ways of "opening" (low-level like fsspec, and high-level like xarray) seem necessary and sufficient, and making one a method and one a top-level entry-point emphasizes the difference.

[chuckwondo]

I like the idea of simplifying things. Here are some obstacles that initially come to mind regarding what you've outlined above:

• Users that already have a collection concept ID or a collection name-version pair (or other unique "key" for a collection) have no need to first produce a collections result before producing a granules result, so this would seem to be counter toward your idea of simplification. Even if we can manage to avoid a collections query/request, this would still be at least slightly less simple, but I don't think it's out of the question. We just need to carefully design the approach.
• This becomes perhaps too tricky if we want to fold in search_services under the singular search umbrella because I don't think we need to first search for collections (I would have to check the docs) in order to find services. Further, even if we did, the "filter" part becomes tricky because an umbrella "filter" function would need to support all possible query parameters, so it would be difficult, if not impossible, to help the user know that if they are searching for granules, they can specify a value for "campaign" (for example), but cannot specify such a param value when searching for services. However, I would question the value of retaining search_services anyway. Perhaps that doesn't belong in earthaccess and should be relegated to python_cmr (and if so, this would perhaps be reason to keep the 2 libs distinct, rather than having earthaccess subsume python_cmr). If we remove search_services, perhaps this is no longer a concern.