Bard14 Spectra

[LishaRamon]

Will be ingesting converted spectra from bard14.

Info from issue:
"Spectra (17) converted in #365. Now we need to upload and ingest.
bard14.zip

Filenames have spaces in it. Not sure if that will be thing or not.

Spectra uploaded to: https://bdnyc.s3.amazonaws.com/SpeX/Prism/
AWS converted spaces to plus signs ('+') in the filenames."

Link to relevant issue: Closes #366

For data ingests:

• includes script used for ingest
• includes modified JSON files
• Add new tests
• Update the Versions table

[LishaRamon]

https://docs.astropy.org/en/stable/io/fits/index.html

Not sure how to read in a folder into data for python to read. Looking up ideas of converting fits and png files into csv data then ingesting them (via loop)

[kelle]

You can use the pre-existing google sheet. Check out lines 8-13 in this script: scripts/spectra_convert/convert_bard14_spectra.py

[LishaRamon]

Is this the link to the sheet you mean? "https://bdnyc.s3.amazonaws.com/SpeX/Prism/" It's broken/non existent to me.
Update: it's in the convert spectra.py file, nevermind got it

[kelle]

As we discussed, use the google sheet.

[kelle]

In this particular case, since we are updating the spectra entries and not adding entirely new ones, we can't use this function.

Suggested change

	# Ingest SPECTRAL TYPES, loop through data
	ingest_spectra(db,
	sources = "Source",
	spectrum= "Spectrum",
	original_spectrum= "Original Spectrum",
	regimes = "regime",
	telescope= "telescope",
	instrument= "instrument",
	mode= "mode",
	observation_date= "observation date",
	spectrum_comments= "spectrum comments",
	spectrum_reference= "spectrum reference",
	ra= "ra",
	dec= "dec",
	aperture= "aperture",
	raise_error=True,
	search_db= True,
	)

[kelle]

Instead of using the ingest_spectra function, I think we want to do use the update method of astrodbkit2 to update the fields. https://astrodbkit2.readthedocs.io/en/latest/#updating-data

• Update the spectrum field to be the new .fits file
• Update the original_spectrum field to be the old .txt file.

Do this row-by-row inside the loop.

[kelle]

This is an extra loop. You already have the loop starting on line 38.

[kelle]

The only fields that need updating are the spectrum and original_spectrum so you don't need any of these other fields.

Suggested change

	regime_value = entry["regime"]
	telescope_value = entry["telescope"]
	instrument_value = entry["instrument"]
	mode_value = entry["mode"]
	observation_date_value = entry["observation date"]
	spectrum_comments_value = entry["spectrum comments"]
	spectrum_reference_value = entry["spectrum reference"]
	ra_value = entry["ra"]
	dec_value = entry["dec"]
	aperture_value = entry["aperture"]

[kelle]

These fields don't need updating.

Suggested change

	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.regime == regime_value)\
	.values(regime = regime_value))
	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.telescope == telescope_value)\
	.values(telescope = telescope_value))
	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.instrument == instrument_value)\
	.values(instrument = instrument_value))
	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.mode == mode_value)\
	.values(mode = mode_value))
	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.observation_date == observation_date_value)\
	.values(observation_date = observation_date_value))
	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.comments == spectrum_comments_value)\
	.values(comments = spectrum_comments_value))
	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.reference == spectrum_reference_value)\
	.values(reference = spectrum_reference_value))
	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.ra == ra_value)\
	.values(ra = ra_value))
	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.dec == dec_value)\
	.values(dec = dec_value))
	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.aperture == aperture_value)\
	.values(aperture = aperture_value))

[kelle]

Suggested change

	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.source == source_value)\
	.values(source = source_value))
	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.source == source_value)\
	.values(spectrum = spectrum_value))

[kelle]

This is the magic sauce. Study this command. You'll need another command like this to update the original_spectrum field.

[kelle]

This is the right way to do the loop.

Suggested change

	# for row in bard14_table[0:19]:
	# for row in bard14_table[0:19]:

[kelle]

This doesn't work. update_all_spectra is the name of the function....it's not a list or table which you can loop over.

Suggested change

	for entry in update_all_spectra:
	for entry in update_all_spectra:

[kelle]

This is not quite right. It's trying to find spectra files that are set to the source name.

Suggested change

	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.original_spectrum == source_value)\
	.values(original_spectrum = original_spectrum_value))
	conn.execute(db.Spectra.update()\
	.where(db.Spectra.c.source == source_value)\
	.values(original_spectrum = original_spectrum_value))

[kelle]

Not sure why you added the _table bit. The text here needs to correspond exactly to the name of the table you're trying to modify.

Suggested change

	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.c.source == source_value)\
	.values(spectrum = spectrum_value))
	conn.execute(db.Spectra.update()\
	.where(db.Spectra.c.source == source_value)\
	.values(spectrum = spectrum_value))

[kelle]

Don't need this. The code on 69-71 updates the spectrum field.

Suggested change

	conn.execute(db.Spectra_table.update()\
	.where(db.Spectra_table.spectrum == spectrum_value)\
	.values(spectrum = spectrum_value))

[kelle]

Once you get the loop working, I also want to check that the spectra links are valid. Here's some code to get you started.

import requests
request_response = requests.head(spectra_value)
            status_code = (
                request_response.status_code
            )  # The website is up if the status code is 200

[kelle]

This is the line which actually checks if the URL is valid. It currently only checks the spectrum_value URL. It is not checking the original_spectrum.

[kelle]

This status_code corresponds to the spectrum_value link, not the original_spectrum_value link.

[kelle]

Now this code only checks the original spectrum and not the spectrum. line 55 overwrites the status_code variable in line 54. You need two different variable names. and you need to check them both.