Toast 3 Work in Progress #369

Hybrid pipeline

* Tweak tolerances for detdata equality tests. * Base tolerances on the data range.

* Add option to write co-add in single precision * Fix issue with applying the scaling factor

* Fix units in demodulated PSD * Reset all samples interval

… more process than devices

Code modification introduced for benchmarking

Fix JAX multi process/nodes

…lization on GPU

…tack usage. (#653)

Fixed offset kernel bug

…cking

Kernel timing

* Fully functional implementation * Fix issues with the file names

* New omp target update PR to replace #603 * Port over target memory handling of null buffers * Port over updated kernels and helper functions * Implement noise weight kernel * Implement scan map openmp kernel * Support runtime memory pool, disabled for now * Run format_source.sh * work in progress on unit tests * Move noise weighted map debug print to before return of data to GPU * Small cleanups to tests, stokes kernel, and data reset * Run format source * Add omp target guards around reset * Add missing omp target guards * Small omp kernel changes and cleanups to allow unit tests to run * In the benchmarks, by default, overwrite the input TOD with the amplitude-subtracted one to save memory.

…array

JAX related fixes

* Debugging and testing omp target offload: * Restructure offset template kernel to avoid additional team private variable. Add reset method to Amplitudes class. * Clean up accelerator use / wrapping in DetectorData object. * Add a unit test for the paradigm of wrapping raw host buffer with jax mutable array. * Tweak json benchmark format * Disable accelerator use when scanning maps off disk * Set omp target default device at initialization * Guard the call to omp_set_default_device * stokes_weights_I_jax is now jit compiled * Updated perlmutter nvhpc config * When using jax outside of slurm, initialize with localhost * fixed 2D weights in build_noise_weighted * added missing use_accel to test * fixed gpu datamovement in test * fixed gpu datamovement in test * fixed restrict test and data access * removed temporary test skip * moved accel_delete call inside observation_data's clear to take into account the fact that device data might be in data or raw * moved accel_delete insideobservation_data's _allocate to take device data being elsewhere than raw into account * removed wrap_raw from _allocate to reduce useless back and forth * removed wrap_raw from change_detectors * added a zero_out argument to observation_data._accel_create to be able to better time creation * Add zero_out argument to PixelData._accel_create(). Remove unnecessary branch for reset of jax array. * Pass kwargs in AcceleratorObject create * Run format_source.sh --------- Co-authored-by: nestor <[email protected]>

JAX: recycling memory

* fixing the detset error * adding a test for common mode * running format source * Add support for user-generated list of common mode detectors --------- Co-authored-by: GIUSEPPE PUGLISI <[email protected]> Co-authored-by: Reijo Keskitalo <[email protected]>

* support locking the azimuth range * Add SPECIAL tag for skipped schedule entries

* Allow writing out the noise-weighted observation matrix * Remove unnecessary messages from co-add * Add script to co-add obs matrices * Add a compiled kernel for co-adding observation matrices * Skip zeros in observation matrix calculation * Remove debug statements * Fix warning about map units * Fix index error * WIP * add unit test for noiseweighted matrices and fix issues that were found * Remove obsolete debugging statements * Fix a deadlock, add an explicit barrier to avoid a race condition in the unit test * Workaround for serial cases

* Update default StokesWeights operator which generates the elements of the pointing matrix: * Update sign convention to match accompanying document (to be added to documentation) given our recent formalization of the focalplane coordinate frame. * When using a HWP, require per-detector offsets in the focalplane table describing the fixed angle from focalplane frame X-axis to detector frame X-axis. * Correctly handle the cases of fixed HWP angle versus no HWP. * Add unit tests that cover several permutations of I/Q/U sky pixel values, fixed HWP angles, and detector orientations. * Small consistency fix for poles when doing quat to ISO conversion. Fix conviqt operator and tests. * Fix jax stokes_weights kernel. Address review comments. Additional changes made to clean up accelerator use: * Helper functions for printing OpenMP target buffer table * Clean up duplicate code in Data accelerator movement * Clean ups to DetectorData accelerator movement * Add use_accel argument to Calibrate operator, in analogy to mapmaking operator * Clean ups to TemplateMatrix accelerator use * Introduce new temporary object specification along with requires / provides. Useful for operators calling other operators for a subset of their products * Fix Pipeline logic tracking staged and unstaged data for hybrid host / accelerator operator case * Add new unit test for Offset template use within an accelerator enabled pipeline * Run format_source * Remove temporary object concept from operator * Clean up dependencies so that detector data which is optionally created by an operator is listed in requires(), so that it is moved if it exists.

* Collection of improvements for working with real ground data: * Add a new operator (AzimuthIntervals) which uses smoothed versions of the azimuth velocity and acceleration to detect azimuth scan patterns and define the typical intervals. * Add support for per-detector flags that apply to an entire observation. These can be set / updated and use the same masks as per-sample detector flags. These flags can now be used to optionally control selection of local detectors for processing. * Add new operator (FlagNoiseFit) which uses the analytic fit to the noise estimate to flag detectors for an observation. The current implementation can flag outliers in both the estimated NET and knee frequency values. * Visualization: small fixes to ipython widget and improvements to WCS and noise estimation plotting functions. * Set the SimGround bitmask to enable turnarounds in the unit tests * Allow optionally running with a single process group in the unit tests, even if using >= 2 processes. * Pull out duplicate code to compute the detector scan range for and observation. Use this common function in atmosphere simulation and WCS projection autoscaling. * When fitting an analytic 1/f model to estimated noise PSDs, allow overriding the frequency range that is considered the white noise plateau. * Address review comments

* Bump release tag * Enable wheel tests * Disable normal tests for wheel debugging. Disable obs matrix combination test in wheel build environment. * Skip all filterbin tests when CIBUILDWHEEL is set * Re-enable normal tests

JAX bugfix

Redesigning OpenMP target offload loops

c.f. mpi4py/mpi4py#34

* Small cleanups for building with LLVM-17: * Restore is_device_ptr clauses and formatting removed in #676 * Update test scripts to support llvm compilation * Remove debugging.

* Fix the simulated focalplane layout spacing. * Clarify that the angular "width" corresponds to the spacing of extreme pixel location along a specific axis. * Fix off-by-one to match this specification. * Add new rhombus_hex layout function as a unification of downstream tools into a central location for testing. * Ensure that all layout functions also propagate the detector gamma angle in the returned detector properties. * Add pixel centers to unit test plots to verify correct layout. * Small tweaks to rhombus hex layout.

such that `python -m toast.scripts.toast_ground_schedule -h` would work or we could do ```py from toast.scripts.toast_ground_schedule import main main(...) ```

Small perf improvement for the JAX Healpix kernel

* Enable wheel tests prior to tag * Fix typo * Fix typo in osx deps install * Attempt to fix suitesparse build for macos wheels * Build only the subset of suitesparse that we need. Update cibuildwheel version. * Disable wheel tests and restore normal tests. * Update our cmake checks to use the modern python3 macro * Update bundled versions of pybind11 and googletest. * Add minimal pyproject file and update Manifest to remove warnings. * More cleanups to MANIFEST.in and setup.py

* Debugging sporadic failure of copying build products into wheels * Update Python3 cmake hint in setup.py * When finding python, set policy to new to use first found interpreter. Also require only Development.Module component. * Restore normal tests

…ation

Updating JAX

* Add compress_detdata to save_hdf5 operator * Add precision argument to compression as an alternative to quanta * Add unit test for FLAC compression precision * Append obs_ to datadump filenames * Do not try verify partial fields that do not exist * Update default pattern * Force 64bit input to compiled kernel * fix typo * Add option to return data with the compression errors * Add support for float32 data * Retain the TOD data type when compressing/decompressing in place * Fix failing unit test

* Attempt to get wheel builds finding static libFLAC. * Enable wheel tests on push * If the libFLAC version cannot be determined, assume it is new enough and use it. * Restore testing workflow

* Implement a new mapmaking template for periodic signals This adds a new template that regresses components of detector data that are correlated with a shared data field whose values are periodic (HWP angle, Azimuth, etc). The values of the periodic field are binned and a different set of template amplitudes are solved for each detector and each observation. Additional changes: * Unit test for fake HWPSS removal in satellite and ground simulations. * Unit test helper for healpix map plotting, similar to the helper function for WCS plotting. * Added functions to the Offset template to support writing and plotting solved amplitudes. * Small improvement to noise model fitting. * Run format_source.sh * Fix typo * Fix another typo and disable plots in CI runs

) * Extract HDF5 observation metadata loading into a separate function * Run format_source

* Fix flagging of noise model fits * Some additional comments * Add comment for error helper function

* Add support for arbitrary field separators * Fix cooler cycle target * Fix silly bugs * Add unit test; fix schedule parsing * Better unit test

…700)

* Remove memreport that was causing deadlocks; allow user to disable writing binned maps * Handle situations where path contains multiple instances of the string 'map' * Make test more robust * Allow user to set the multiplicity * int32 will overflow in large sparse matrices * Better handling of obsmat suffix * Fix unit test

* Add a unit test for a corner case met in DC0 * Fix numpy kernel * Refactor healpix and stokes weights operations: * Move all current bundled healpix ops into the same source file as the PixelsHealpix code so that all device kernels are in the same compilation unit as the python wrappers. * Add full unit test coverage for all bundled operations, comparing to the equivalent functions in healpy. * Remove dead versions of the code from the deprecated libtoast internal library. * Remove dead versions of the stokes weights code from the libtoast library and call the modern kernels in all unit tests. * In the bundled ang to pix operations, perform range reduction to 0-2PI and also round phi angles to zero if they are close to machine precision from zero. This fixes the issue that was the original motivation for this PR. * Update Stokes pointing weights to match latest document on polarization response. * Remove stale versions of the stokes weights code and move all helper functions closer to the relevant operator kernel. * Fix the use of the "cal" trait of the StokesWeights operator, which was designed to be the name of the observation key to use for a detector calibration dictionary. This change propagated into the kernel code. Additionally, add a small fix to the spt3g unit tests. * Address review comments, run format_source --------- Co-authored-by: Reijo Keskitalo <[email protected]>

* Make a general directory for packaging and move wheel scripts there * Begin refactor of dependency compile scripts. Add conda helper scripts * Port wheel scripts to use new modular dependent install scripts * Enable wheel tests * Fix typos * More typo fixes on macos build * install cmake with homebrew * More fixes to build scripts * Pass shared library suffix to dependency recipes * Clean up platform scripts * Work on high-level setup scripts for nvhpc * Add missing files * Fix ground benchmark segfault. Several fixes to NVHPC build scripts. * More small fixes from perlmutter tests * Add readme for platforms directory * Fix test runner install * Small tweaks to install scripts * Add note about GLIBC symbols on perlmutter. * Update build scripts to support use of a virtualenv * Remove nvhpc mpi specialization * Try static build. Disable timing tests on CI. * Add venv configs for gcc * Temporarily remove use of omp schedule(static) and simd pragmas, for debugging nvhpc threading library. * Test LLVM compilation. Restore lines that were commented out for debugging. * Add build configs for LLVM * Add configs for dynamic linking with llvm * Small changes to both LLVM and NVHPC configs. * Small tweaks to make nvidia compilers happy. * Restore omp simd clauses commented out during debugging

* Bump release string and test wheels * Restore tests * Fix PerturbHWP deadlock when raising exceptions

* take cal dimension into account for stokes_weight * fixed ang computation in stokes_weights_IQU --------- Co-authored-by: nestor <[email protected]>

* Recognize the healpy column_units keyword * Emulate healpy.read_map() better * Emulate healpy.write_map() better * Clean up write_healpix method

* Check error code * Support filtering single precision signal * Use more robust linear regression * Better handling of error conditions

* Handle overdefined problems * Update JAX kernel with the changes to the C++ kernel * Update Numpy kernel with the changes to the C++ kernel

* Initial, working implementation * Small tweak to make interpolation more robust

…o set trait args (#713)

* Add working configs for perlmutter with target offload * Work on README * Update platform readme

* Add new capabilities to the Bandpass class: optical loading and K->W conversion * More sensible units * Remove unnecessary import

Currently, ```log error: invalid path 'src/toast/aux/benchmarks/stats.json' error: invalid path 'src/toast/aux/weather/LMT.h5' error: invalid path 'src/toast/aux/weather/atacama.h5' error: invalid path 'src/toast/aux/weather/convert.py' error: invalid path 'src/toast/aux/weather/south_pole.h5' ``` Because AUX is reserved in Windows. This PR propose replacing aux by _aux using the following code: ```bash FROM=aux TO=_aux sed -i "s/$FROM/$TO/" src/toast/CMakeLists.txt find src/toast -name '*.py' -exec sed -i "s/\"$FROM\"/\"$TO\"/" {} + git mv "./src/toast/$FROM" "./src/toast/$TO" ```

Jax interval map

* Implement consistent handling of flags * More cleanups, run format_source * Pass amplitude flags into Offset template kernels. * More work on flagging consistency * Update flag masks to be inclusive for filtering and more conservative for binning * Remove debugging * Add more debugging plots * Restore the artificial flagging used in the obsmatrix unit test. * Restore another check * Update filterbin flag defaults based on offline conversation. * Fix handling of flagged detectors in noise estimation. Better support for disabled templates in template matrix. * Add detector cutting to all unit tests by default. Cleanup resulting problems due to assumptions about looping over local detectors. Add extra debug plot support to offset template. * Debugging test failures * Ensure that the noise weighting operator always sets output detdata units, even if not processing detectors from some observations. * Update platform scripts from llvm-17 to llvm-18. * Address some review comments * Split out the per-detector flag mask as a trait separate from the per-sample flag mask * Many small fixes, address review comments. * Remove redundant memory clearing. Fix Offset template band-diagonal preconditioner. Fix timing imports to point to timing rather than utils module. * Revert name of demodulation stokes weights. Fix typo in FlagNoiseFit. * Comment out debug statements * Fix stale comment

…ed kernels.

Jax templates

* Fix parallelization problems in noise estimation * Small fixes for self.view=None * Some small fixes, discovered with unit tests and batch jobs (#730) with 8 or 16 processes per group: * When duplicating an observation, also duplicate per-detector flags. * When redistributing an observation, reset per-detector flags before setting. * Make the common mode removal an option (default True) prior to noise estimation. This is useful if the data has already had the common mode removed prior to this operator. * When high-pass filtering data that is distributed by time slices there may be some chunks that are fully flagged. This should not be an error. Instead, the flagged timestream is set to zero. * When communicating overlaps, ensure that the span used (lagmax + half_average) is less than the number of samples. * Ignore flagged detectors in poly2d fit --------- Co-authored-by: Theodore Kisner <[email protected]> Co-authored-by: Ted Kisner <[email protected]>

* The current configuration system suffers from some challenges, including fragile / special-case parsing of container traits. This work aims to make this area of the code robust and more future-proof: * Move config tools into a submodule, extract some things from traits.py. * Support arbitrary nested containers, including parsing of those and None values from the commandline. Expand unit tests to included nested containers. * Dedicated set of routines for translating back and forth from trait values and strings. * Separate file-format code more cleanly from code that just works with the internal config dictionary format. * Add support for YAML config files, along with existing JSON and TOML formats. * Expand the `toast_config_verify` script to support loading and migrating old config files to new ones (or for converting between supported formats). * Keep support for loading old config files, while printing warnings telling the user to update their configs. * Address review comments. Run format_source.sh

* Remove cancel-workflow github action and use concurrency instead * Enable wheel tests * Fix SaveHDF5 operator to allow volume=None. * Fix typo

* Remove default flag bits where intervals are more natural * No separate flags for scan direction; raise unstable scan rate flags for turnarounds * Use intervals in place of shared flags to identify left and right-going scans * convert another unit test to use intervals

* Guard against empty lists of detectors * Tweak the atmospheric simulation to use the concentric cones more effectively * Fix and speed up atmospheric unit tests * Remove commented lines

* Bump pre-release version * Small C.I. updates: - Update versions of github actions. - Switch to using our own miniforge bootstrap (rather than github action), to have more control over the environments.

* When parsing config files, handle compatible types Due to the way we are writing TOML and YAML files for user-friendly editing, the true type of traits loaded from those formats can be ambiguous. This work adds checks to allow assignment from compatible types when merging the values loaded from config files with the current state of trait properties. * More workarounds for backwards compatibility with old configs. * Fix typo in toast_config_verify * Fix config updates in toast_config_verify

* Implement a mapmaking template for HWP synchronous signal - Implement truncated Fourier series technique based on C++ code developed for real-time filtering in kst in 2009. - Add unit tests for low-level math functions and a synthetic scan strategy without atmosphere. - Format source with latest version of black (24.1.1) * Move fake HWPSS test generation to helpers. Expand hwpss filter operator tests. * Work on demodulation unit tests * More diagnostics for some tests. Change unit test sample rates and HWP rate for more realism. * Run format_source.sh * Debugging weights * Missing data objects in a particular observation is not an error. * Allow hwpfilter to ignore observations without a HWP. Add option for debugging plots in Fourier2D template * Address review comments

* Be smarter about selecting config loader * Small fix to config loader

* Make AzimuthIntervals more robust to initial and trailing unstable pointing. * Make removal of short intervals optional

* Implement an observation matrix class to simplify use of observation matrices * isort imports * Sort imports

* Fix flagging unfiltered data * Only first process in the column should update shared flags

* Flatten list of lists * Pipeline must execute even if there are no local detectors

* Restore HWP Stokes model changes This restores the HWP Stokes response to the model before #736. The changes in that PR were based on observations of a source with known polarization, but resulted in breakage of common-sense tests with discrete detector / HWP orientations. Those polarized source results should be revisited. * Fix detection of test failures * Base groundfilter test criteria on simulated noise level * Fix import. Fix absolute tolerance in baseline comparison. * Fix absolute imports and missing import from obsmat.py * Bump requirements to pshmem 1.0.5, which has fix for macos. * Try to increase the shell limit on number of open files when running tests on macos. * More attempts to increase open file limit on macos * More attempts to increase open file limit on macos again * Bump pshmem requirements. * Speed up some unit tests. Fixes to non-default stokes weights kernels.

* Add entrypoint for plotting, move plot function from test helpers to vis. * Add entrypoint for healpix plotting as well

* Increase sat sim verbosity * Add reporting * Fix typo * Indicate the correct reference frame * Fix coordinate system conversion during pointing expansion * Add coordinate system support to satellite simulation * Improve reporting * Clean up code * Refactor map writing into a separate method * Fix unit test failure * Refactor mapmaker for better readability * Run source formatter * Refactor template solver * Add --sample_rate and --fsample to toast_sim_satellite.py * Add gpu staging to coordinate rotated boresight (#750) * Eventually, this could be moved to a dedicated operator which creates rotated boresight pointing. * For now, this is handled by the pointing_detector operator, and the rotated boresight is created if needed, and moved to the appropriate device. * Restore original defaults --------- Co-authored-by: Theodore Kisner <[email protected]>

* Implement an operator for quick-and-dirty white noise modeling * Add missing file

* Small changes for plotting * Run format_source.sh

* Do not redistribute data unless computing cross spectra. When estimating the noise PSDs, computing autospectra does not require redistribution. Also apply a small fix to one of the plotting routines. * Run macos tests on x86_64 conda env

* Sensible default values for Weather * Refine atmosphere * Derive the scan range when not available * Add the left- and rightgoing intervals * Add operator to append weather model to existing observations * Raise an error when no submaps are defined * More robust short interval identification * Add a simple deglitch operator * Make outlier detection more intuitive * Add jump correction module * Add regression mode to common mode filter * Improve AzimuthIntervals and other small fixes. (#756) - Modify the AzimuthIntervals operator to more robustly exclude false "throw" intervals. Instead we first detect the stable pointing periods and then find the exact turnaround sample between those (and raise an exception if there is more than one). This does mean that beginning of the first throw and the end of the last throw are truncated to the stable scan boundary. - In the mapmaker, keep the noise weighted map if `keep_final_products` is True. - In the mapmaker, if the binner is configured to compute and save full detector pointing, use that option when initially computing the pixel distribution. This avoids computing the pointing twice. - When scanning from a map in the template solver, if we already have full detector pointing, then run that over all detectors. - Fix typo in mapmaker from recent refactor. * Revert unnecessary changes to atmo sim verbosity * Fix deglitch provides * Fix jumpcorrect provides --------- Co-authored-by: Theodore Kisner <[email protected]>

@gabrielecoppi

* Refactor of WCS pixelization operator - Move the application of source centering in the projection to a separate helper function in `pointing_utils.py`. Thanks to @gabrielecoppi for identifying this fix. Optionally use this new function when computing the scan range for autoscaling. - In `PixelsWCS`: - Add a new general class method that computes the WCS parameters. - Add support for SFL projection. - Allow projection traits to be changed in any order and only recompute the WCS if needed when exec() is called. - Default to a single submap, which is the most efficient choice for the common case of data distributed by detector and many observations co-incident on the sky. - In the PixelsWCS unit tests: - Ensure projection and plotting works for every supported projection type with both fixed parameters and autoscaling. - Test mapmaking in both normal mode and with source-centered projections in RA/DEC and Az/El. - In `plot_wcs_maps`: - Set the figure size based on the DPI and the actual size of the image in pixels. - Set the unhit pixels to gray. - Allow specifying the color map, and default to one of the perceptially uniform ones. * Fix other unit tests * For local Az/El coordinate frame, use TLON/TLAT in the CTYPE. Thanks to @gabrielecoppi for the suggestion. * PixelDistribution global2local is now always created * Fix typo

* Run format_source with latest versions of tools * Bump pre-release version

* Work on fixing wheels. Build on macos-12 runner to have native x86_64 host. Bump vendored openblas version. * Disable tests temporarily * Restore cmake change on macos * Attempt to use MACOSX_DEPLOYMENT_TARGET on macos wheel builds * Disable all flac programs when building a wheel to avoid linking to homebrew versions of libintl. Upload wheel artifacts named after the python version and arch. * More attempts to fix libflac compilation for macos wheels * Another attempt to uninstall conflicting packages on macos wheel builds * Restore workflow defaults

* Check the correct communicator * Do not define nonexistent traits

* Add a sidereal targeting mode * Improve help string

* Do not try to invert 0-dimensioned covariance matrices * Fix serial mode

* Update simple sim_ground workflow (#1) * update operator names and inputs * try to infer weather from schedule if not provided * standardize azimuth interval before checking against simulation bounds

* Add processing bit to non-science shared mask * Fix unrelated typo in center lonlat offset

* Test wheel builds * Remove deprecated use of get_cmap * Restore normal test workflow

* Extend scan-synchronous signal to have polarization * Fix logic error

* Various small fixes - Fix deadlock in 1D polyfilter if a process has no good detectors - When flagging noise model outliers, iterate until no further detectors are cut. - In the template classes and Amplitude class, handle the case where some processes have no good detectors. * Fix typos

* Extend scan-synchronous signal to have polarization * Fix logic error

#776) * Add option in the detector pointing operator to deflect the pointing based on the HWP angle * Make toast the first import * Add HWP phase and unit test

* Various fixes and features for dealing with realistic data. * Move HWPSS utility functions into a separate source file * Add a new CalibrateDetectors operator which takes a dictionary of factors to apply per observation. * Change detector timeconstant deconvolution to use serial rather than batched FFTs by default. This reduces memory footprint in the common case where most parallelism comes from MPI. * Add option to AzimuthIntervals to also cut extraneous long intervals * Support both PDF and PNG image formats for plots. * When demodulating data, also propagate per-detector flags. * Fix deadlocks caused by logging barriers in noise estimation. * Several small fixes * Re-enable flags in hwpfilter test * Address review comments

…779) * Modify deflection code so that the overall detector polarization direction stays fixed (i.e. the deflection is a translation) * Add a unit test which makes plots of the detector pointing * Small fixes to the plotting utility * Small fix for case where quaternions are constructed from an array of axes.

* Update internal RELEASE file * Fix bundled package scripts so that libflac build respects the shared and static configuration setting

The validation for the nbin_psd trait was assuming an integer, but it could be None

…array. (#785) Wrap matplotlib.cbook.flatten calls in a list before constructing an array, since these are generator expressions without a fixed length.

* First functional implementation * Fix parallelization issues * Remove save_pointing * Remove unused variables * More consistent used of the observation reference * Remove unused trait

* Even super short observations need at least two SLERP points * If the user does not want the map, don't bin it. * One more case when binning is required

* Add method to sort observing schedule by RA * Replace and with

* Various fixes and features for dealing with realistic data. * Move HWPSS utility functions into a separate source file * Add a new CalibrateDetectors operator which takes a dictionary of factors to apply per observation. * Change detector timeconstant deconvolution to use serial rather than batched FFTs by default. This reduces memory footprint in the common case where most parallelism comes from MPI. * Add option to AzimuthIntervals to also cut extraneous long intervals * Add new operator to simultaneous remove a Maxipol style HWPSS template and build relative calibration factors from the 2f magnitude * New operator for estimating HWP synchronous signal. This work introduces a new operator that can model HWPSS in various ways, and optionally estimate the relative gains between detectors based on the 2f harmonics. Features include: - Detection and flagging of samples with a stopped HWP. - Model coefficients can be estimated on the whole observation, on fixed-length chunks, or on pre-defined intervals. The chunk-wise model coefficients are then smoothly interpolated over the observation. - Model can optionally include a time drift separate from the chunking. - The 2f harmonic is used to estimate the relative gain between detectors, either as a fixed value per observation or as a continuous calibration timestream. This can be used to flag outlier detectors and generate calibration tables / timestreams for application. - Includes proper treatment of flagged samples in normalization of model coefficients and covariance. - Extensive optional debug plots. * Fix unit tests * Fix hwpss destriping template unit test * - Add helper methods for determining if an observation is distributed purely by detector or sample. Use these everywhere in the code that currently does this manual check. - Extend flagging of stopped HWP to also include acceleration / deceleration periods. Add unit test for this functionality. - Generate an error when all fit chunks fail. - Handle case where only one chunk fails. * Fix typo * Remove support for python-3.8, which has reached end of life. * Bump requirements to python-3.9.

Fix not using the `det_mask` input value when selecting detectors.

Change default bit mask to `defaults.det_mask_invalid`. `defaults.det_mask_nonscience` exclude turnaround, it is valuable to deglitch or jump correct the turnarounds as well.

scipy.signal.convolve will choose the fastest convolve method (direct or fft) based on scipy.signal.choose_conv_method result.

Add missing while loop, that iteratively find jumps.

Move `len(mytoi)` outside of loop to avoid repeated calculation.

…efault bit mask (#794) * Fix not using given `det_mask` bit mask Fix not using the `det_mask` input value when selecting detectors. * Change default bit mask Change default bit mask to `defaults.det_mask_invalid`. `defaults.det_mask_nonscience` exclude turnaround, it is valuable to deglitch or jump correct the turnarounds as well.

Jump finder fix

* Update bundled pybind11 and random123 to latest versions for better compatibility with clang++ on arm64. * Add __init__.py files to data directories to silence warnings. * Use importlib.resources.as_file() instead of pkg_resources for compatibility with python-3.12. * Build wheels with numpy-2.0.x, which is backwards compatible at runtime with numpy-1.x and also compatible with numpy-2.1.x. * When building suitesparse for wheels, remove patch and use the cmake system to enable only cholmod and build everything (rather than use the archaic Makefiles directly). * Lift runtime requirements on suitesparse and numpy since we are now compatible with the latest versions. * Bump versions of vendored OpenBLAS and suitesparse. * For wheels on macos, build our own openblas rather than use libscipy_openblas, which has strange symbol name mangling that does not seem to work with clang++. * In the unit test workflow, add python-3.12 tests and also run tests on macos arm64. * In the wheel test and deploy frameworks, add python-3.12 and macos arm64 to the build matrix.

Commits on Mar 13, 2023

Bump pre-release

tskisner committed Mar 13, 2023

Configuration menu

View commit details

Copy full SHA for 87479b7

Browse repository at this point

Copy the full SHA

87479b7 View commit details

Browse the repository at this point in the history

Commits on Jun 6, 2023

Address PR review

keskitalo committed Jun 6, 2023

Configuration menu

View commit details

Copy full SHA for 7401963

Browse repository at this point

Copy the full SHA

7401963 View commit details

Browse the repository at this point in the history

Commits on Dec 8, 2023

cleaned code

nestor committed Dec 8, 2023

Configuration menu

View commit details

Copy full SHA for 768bde3

Browse repository at this point

Copy the full SHA

768bde3 View commit details

Browse the repository at this point in the history

Commits on Dec 9, 2023

small improvements to mapping function

nestor committed Dec 9, 2023

Configuration menu

View commit details

Copy full SHA for cf2d3cc

Browse repository at this point

Copy the full SHA

cf2d3cc View commit details

Browse the repository at this point in the history

Commits on Dec 19, 2023

better timing (#722 )

keskitalo authored Dec 19, 2023

Configuration menu

View commit details

Copy full SHA for 1b17c7e

Browse repository at this point

Copy the full SHA

1b17c7e View commit details

Browse the repository at this point in the history

Commits on Mar 21, 2024

Bump release

tskisner committed Mar 21, 2024

Configuration menu

View commit details

Copy full SHA for 004d5bf

Browse repository at this point

Copy the full SHA

004d5bf View commit details

Browse the repository at this point in the history

Toast 3 Work in Progress #369

Are you sure you want to change the base?

Toast 3 Work in Progress #369

Commits on Mar 2, 2023

Commits on Mar 3, 2023

Commits on Mar 9, 2023

Commits on Mar 10, 2023

Commits on Mar 13, 2023

Commits on Mar 21, 2023

Commits on Mar 22, 2023

Commits on Mar 23, 2023

Commits on Mar 31, 2023

Commits on Apr 2, 2023

Commits on Apr 4, 2023

Commits on Apr 5, 2023

Commits on Apr 6, 2023

Commits on Apr 7, 2023

Commits on Apr 8, 2023

Commits on Apr 10, 2023

Commits on Apr 14, 2023

Commits on Apr 16, 2023

Commits on Apr 17, 2023

Commits on Apr 18, 2023

Commits on Apr 24, 2023

Commits on May 24, 2023

Commits on May 25, 2023

Commits on May 26, 2023

Commits on Jun 5, 2023

Commits on Jun 6, 2023

Commits on Jun 16, 2023

Commits on Jun 21, 2023

Commits on Jun 23, 2023

Commits on Jul 5, 2023

Commits on Jul 14, 2023

Commits on Jul 15, 2023

Commits on Jul 17, 2023

Commits on Jul 18, 2023

Commits on Jul 19, 2023

Commits on Jul 24, 2023

Commits on Jul 25, 2023

Commits on Jul 26, 2023

Commits on Jul 27, 2023

Commits on Aug 8, 2023

Commits on Aug 9, 2023

Commits on Aug 16, 2023

Commits on Aug 17, 2023

Commits on Sep 13, 2023

Commits on Sep 20, 2023

Commits on Oct 9, 2023

Commits on Oct 10, 2023

Commits on Oct 11, 2023

Commits on Oct 12, 2023

Commits on Oct 17, 2023

Commits on Oct 23, 2023

Commits on Oct 26, 2023

Commits on Oct 27, 2023

Commits on Oct 31, 2023

Commits on Nov 7, 2023

Commits on Dec 1, 2023

Commits on Dec 4, 2023

Commits on Dec 5, 2023

Commits on Dec 6, 2023

Commits on Dec 7, 2023

Commits on Dec 8, 2023

Commits on Dec 9, 2023

Commits on Dec 11, 2023

Commits on Dec 12, 2023

Commits on Dec 13, 2023

Commits on Dec 19, 2023

Commits on Jan 4, 2024

Commits on Jan 5, 2024

Commits on Jan 9, 2024

Commits on Jan 25, 2024

Commits on Jan 26, 2024

Commits on Jan 27, 2024

Commits on Jan 30, 2024

Commits on Jan 31, 2024

Commits on Feb 1, 2024

Commits on Feb 2, 2024

Commits on Feb 12, 2024