refactor pipeline.new_sources.new_sources

[ddobie]

In #730 I fixed a memory leak in pipeline.new_sources.new_sources, but the actual implementation was quite hacky.

As commented in the code, the function should be rewritten to reflect the new behaviour of pipeline.new_sources.parallel_get_rms_measurements. The below section should be incorporated into parallel_get_rms_measurements, and parallel_get_rms_measurements should be renamed to something like parallel_get_new_high_sigma.

vast-pipeline/vast_pipeline/pipeline/new_sources.py

Lines 439 to 472 in 705a815

	new_sources_df = parallel_get_rms_measurements(
	new_sources_df, edge_buffer=edge_buffer
	)
	# this removes those that are out of range
	new_sources_df['img_diff_true_rms'] = (
	new_sources_df['img_diff_true_rms'].fillna(0.)
	)
	new_sources_df = new_sources_df[
	new_sources_df['img_diff_true_rms'] != 0
	]
	# calculate the true sigma
	new_sources_df['true_sigma'] = (
	new_sources_df['flux_peak'].values
	/ new_sources_df['img_diff_true_rms'].values
	)
	# We only care about the highest true sigma
	# new_sources_df = new_sources_df.sort_values(
	# by=['source', 'true_sigma']
	# )
	# keep only the highest for each source, rename for the daatabase
	new_sources_df = (
	new_sources_df
	# .drop_duplicates('source')
	.set_index('source')
	.rename(columns={'true_sigma': 'new_high_sigma'})
	)
	# moving forward only the new_high_sigma columns is needed, drop all
	# others.
	new_sources_df = new_sources_df[['new_high_sigma']]