-
Notifications
You must be signed in to change notification settings - Fork 18
Comparing changes
Open a pull request
base repository: lsst/qserv
base: 2023.7.1-rc3
head repository: lsst/qserv
compare: main
Commits on Jul 19, 2023
-
Fixed a bug in qhttp response handler
The older version of the code was throwing an exception when creating HTTP headers for file sizes exceeding signed 32-bit representation (~2 GB).
Configuration menu - View commit details
-
Copy full SHA for 8c799cb - Browse repository at this point
Copy the full SHA 8c799cbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 790e379 - Browse repository at this point
Copy the full SHA 790e379View commit details -
Add support for the file-based result delivery to worker's CMSD config
Added the corresponding configuration option to the entry-point CLI for ingecting a value of the result folder into the config file of of the worker CMSD. The configuration file has been extended as well to allow serving files from workers via the XROOTD file-based protocol. The SSI provider class has been extened to recognise result files at the local filesystems of workers as valid XROOTD resources.
Configuration menu - View commit details
-
Copy full SHA for ae9a111 - Browse repository at this point
Copy the full SHA ae9a111View commit details -
Refactored and extended worker configuration service
The service is now available to the client code via shared pointer that can be stored and used by classes as needed. The change reduces the number of parameters which are sent around the code. In the new version of the code only the shared pointer to the service is put to where the configuration parameters are consumed. Also added new parameters to support the file-based result delivery, including: a location of the results folder, the number of BOOST ASIO threads to run the QHTTP server at worker, and a selector for the desired results delivery protocol.
Configuration menu - View commit details
-
Copy full SHA for 27ab504 - Browse repository at this point
Copy the full SHA 27ab504View commit details -
Extended Protobuf definition to return file resource locations
Also elminated deprecated protobuf schema and large result attributes
Configuration menu - View commit details
-
Copy full SHA for fab0082 - Browse repository at this point
Copy the full SHA fab0082View commit details -
Major refactoring and code cleanup in the worker query processing cla…
…sses Split wbase::SendChannelShared into the base class and an implementation Did the minor refactoring in naming and using XROOTD/SSI streaming channels These change were required to prepare ground for introducing other "ChannelShared"-alike implementations of the result processing and delivery mechanisms. The refactoring aims at correcting dependencies in a code that creates and sets up worker tasks, so that task initialization was happening completelly within the implementation file of wbase/Task.cc rather than being spread between wbase/Task.cc and wcontrol/Foreman.cc. Altogether this is meant to improve the observability of the code and make it easier to maintain. Also, some minor code cleanup and removal of unused header includes and forward declarations was made. Thould reduce the compilation time of the relevant modules. Got rid of unused members and methods Classes wbase::ChannelShared (used to be wbase::SendChannelShared) and wbase::TransmitData had an obsolete mechanism for setting up result schema in the response messages to be sent to Czar. The current implementation no longer uses that API. The obsilete API was removed to avoid confusions as it wasn't obvious where the actual schema settings were made. Got rid of std:: in the iomplementation files with "using std". Guaranteed and enforced synchronization in private methods that require a lock to be held before calling the methods. In the new code, a cons reference to a lock is passed around contexts (methods) where such lock is required. This compiler-enforced technique has proven to work for large code bases where it would be hard to track a state of the lock by other methods. The only other alternative method would be to do the run-time inspection of the lock at the entrance of a method.
Configuration menu - View commit details
-
Copy full SHA for 7e4dd53 - Browse repository at this point
Copy the full SHA 7e4dd53View commit details -
Implemented writing and serving result files by workers
The new result writer redirects result sets into files at workers. Only the last "summary" message with no rows is sent to Czar for each worker query (regardless if a number of tasks that were required to process the query). Also did the minor refactoring in the error messages Added HTTP server to Qserv worker for serving result files Also made minor fixes and improvements to basic tests of QHTTP made in the Replication/Ingest system code base. Using a configuration option to select the desired result delivery protocol Added support (is configured) for deleting unclaimed result files at the startup time of the worker service and after restarting the Czar.
Configuration menu - View commit details
-
Copy full SHA for 526eaa1 - Browse repository at this point
Copy the full SHA 526eaa1View commit details -
Eliminated dead code in the result merger at Czar
It was the trivial refactpring of the code meant to simplify the code and prepare it for the functional extention.
Configuration menu - View commit details
-
Copy full SHA for 0376450 - Browse repository at this point
Copy the full SHA 0376450View commit details -
Modified Czar to allow pulling result files from workers
The new version of the result merger at Czar would dynamically determine which protocol should be used for pulling results from workers: SSI stream, a file read via the XROOTD file protocol, or a file read via the HTTP protocol. In case of the last two options, the merger would also also utomatically tell a worker to delete the result file upon completion of the merge (including the unsuccessful ones). A choice of the delivery method is based on the optional fileds in the Protobuf messages received from workers. It's up to the workers to decide on what method to select.
Configuration menu - View commit details
-
Copy full SHA for 4260547 - Browse repository at this point
Copy the full SHA 4260547View commit details -
Added a monitor for the results filesystem
Also, extended the Web Dashboard to display the filesystem status and usage statistics.
Configuration menu - View commit details
-
Copy full SHA for 77ffa3e - Browse repository at this point
Copy the full SHA 77ffa3eView commit details
Commits on Jul 20, 2023
-
Configuration menu - View commit details
-
Copy full SHA for fe21aa3 - Browse repository at this point
Copy the full SHA fe21aa3View commit details
Commits on Jul 21, 2023
-
Refactored configuration service of Czar
The service is now implemented as a singleton which is initialized and loaded with the configuration parameters at a time when Czar is being created. The rest of the Czar fetches values of the parameters from the singleton. The refactoring aims at reducing the number of parameters sent around in the code and to improve readability of the code.
Configuration menu - View commit details
-
Copy full SHA for bb80f0b - Browse repository at this point
Copy the full SHA bb80f0bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 38bac57 - Browse repository at this point
Copy the full SHA 38bac57View commit details -
Enforce large result limit at workers when fetching result sets
Note that the enforcement only applies to the file-based result delivery protocol as this mechanism makes no sense when results are streamed to Czar over the SSI protocol. The large result conditions are reported to Czar with the code util::ErrorCode::WORKER_RESULT_TOO_LARGE which would help Czar to identify this specific condition and properly report the one to a user.
Configuration menu - View commit details
-
Copy full SHA for 53b951a - Browse repository at this point
Copy the full SHA 53b951aView commit details -
Reinforced response processing in the result merger
The new code verifies if the summary messages reported for the file-based results don't have any error contexts. Abort result merging should any such contexts be found.
Configuration menu - View commit details
-
Copy full SHA for 77b1680 - Browse repository at this point
Copy the full SHA 77b1680View commit details -
Configuration menu - View commit details
-
Copy full SHA for 951c35d - Browse repository at this point
Copy the full SHA 951c35dView commit details -
Fixed a candidate bug in the MySQL query cancellation
Also, added a logging statement on failures to do so.
Configuration menu - View commit details
-
Copy full SHA for 044b146 - Browse repository at this point
Copy the full SHA 044b146View commit details
Commits on Jul 22, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 655206f - Browse repository at this point
Copy the full SHA 655206fView commit details
Commits on Aug 1, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 7344288 - Browse repository at this point
Copy the full SHA 7344288View commit details -
Configuration menu - View commit details
-
Copy full SHA for c661f0f - Browse repository at this point
Copy the full SHA c661f0fView commit details -
Migrated REST API of the Replication system
The new implementation of the Qserv query search service no longer uses the full-text indexes on the query text. The service now relies on the simpler search methods LIKE and REGEXP. The version number of the API is now 23.
Configuration menu - View commit details
-
Copy full SHA for f8e1c8c - Browse repository at this point
Copy the full SHA f8e1c8cView commit details -
Web Dashboard: migrated to follow changes in the REST API
Changes affect the search for for the past queries.
Configuration menu - View commit details
-
Copy full SHA for 4221c20 - Browse repository at this point
Copy the full SHA 4221c20View commit details -
Configuration menu - View commit details
-
Copy full SHA for 94847e6 - Browse repository at this point
Copy the full SHA 94847e6View commit details
Commits on Aug 10, 2023
-
The minor extension made to the MySQL connector
Also added the utility class to handle simple query cases.
Configuration menu - View commit details
-
Copy full SHA for 42b7db6 - Browse repository at this point
Copy the full SHA 42b7db6View commit details
Commits on Aug 17, 2023
-
Moderate refactoring and code consolidation in the worker requests
Simplified error reporting. Reduced code duplication.
Configuration menu - View commit details
-
Copy full SHA for bc0ca76 - Browse repository at this point
Copy the full SHA bc0ca76View commit details -
Configuration menu - View commit details
-
Copy full SHA for 20293e6 - Browse repository at this point
Copy the full SHA 20293e6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 73f72f3 - Browse repository at this point
Copy the full SHA 73f72f3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 62a32be - Browse repository at this point
Copy the full SHA 62a32beView commit details -
Extended Replication system's database API
Added a function for getting info on the ongoing database activities as reported by the query SHOW [FULL] PROCESSLIST.
Configuration menu - View commit details
-
Copy full SHA for 99c7e20 - Browse repository at this point
Copy the full SHA 99c7e20View commit details -
Extended Controller's REST API to report database activities
Added the REST services to report database activities at Czar and Qserv workers. Incremented the version number of the REST API.
Configuration menu - View commit details
-
Copy full SHA for 951df6e - Browse repository at this point
Copy the full SHA 951df6eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 098cf37 - Browse repository at this point
Copy the full SHA 098cf37View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3f550dc - Browse repository at this point
Copy the full SHA 3f550dcView commit details -
Capture MySQL thread identifiers for worker queries
The identifiers are captured for the worker task monitoring purposes. They are used by the Web Dashboard for associating task activities with the corresponding MySQL queries issued by the tasks. The task monitoring page of the Dashboard has been extended to display a value of the identifier (if it was set for the task).
Configuration menu - View commit details
-
Copy full SHA for a502f15 - Browse repository at this point
Copy the full SHA a502f15View commit details -
Refined states of the worker tasks
Introduced an additional state STARTED to differentiate between two events - when a task was started by a scheduler from the actual start time of the corresponding MySQL query. The change is meant to improve the monitoring of the worker tasks. The corresponding monitoring page of the Web Dashboard was modified as well.
Configuration menu - View commit details
-
Copy full SHA for e22ca72 - Browse repository at this point
Copy the full SHA e22ca72View commit details -
Configuration menu - View commit details
-
Copy full SHA for cefbb18 - Browse repository at this point
Copy the full SHA cefbb18View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4971fc5 - Browse repository at this point
Copy the full SHA 4971fc5View commit details
Commits on Aug 24, 2023
-
Fixed a synchronization bug in the helper class
Minor refactoring in the class.
Configuration menu - View commit details
-
Copy full SHA for b3d4bcc - Browse repository at this point
Copy the full SHA b3d4bccView commit details
Commits on Aug 28, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 9aaf64f - Browse repository at this point
Copy the full SHA 9aaf64fView commit details
Commits on Sep 7, 2023
-
Force group and/or user deletion in user-specific build container if …
…there is a clash
Configuration menu - View commit details
-
Copy full SHA for 88eb123 - Browse repository at this point
Copy the full SHA 88eb123View commit details -
Configuration menu - View commit details
-
Copy full SHA for 80fa2b4 - Browse repository at this point
Copy the full SHA 80fa2b4View commit details -
Configuration menu - View commit details
-
Copy full SHA for cf19796 - Browse repository at this point
Copy the full SHA cf19796View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5303ba9 - Browse repository at this point
Copy the full SHA 5303ba9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 80f6786 - Browse repository at this point
Copy the full SHA 80f6786View commit details -
Configuration menu - View commit details
-
Copy full SHA for 33f04ba - Browse repository at this point
Copy the full SHA 33f04baView commit details
Commits on Sep 8, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 3f76000 - Browse repository at this point
Copy the full SHA 3f76000View commit details -
Merge pull request #808 from lsst/u/fritzm/readme-main-fix
Fix master->main problems in README.md
Configuration menu - View commit details
-
Copy full SHA for db15f28 - Browse repository at this point
Copy the full SHA db15f28View commit details
Commits on Sep 14, 2023
-
Moved the function for finding host's FQDN(s) into util/
Extended the contract and the implementation of the function to allow a choice to return a single result or all findings. Migrated the only client of that function that existed before the move.
Configuration menu - View commit details
-
Copy full SHA for 110f6ee - Browse repository at this point
Copy the full SHA 110f6eeView commit details -
Fixed a bug in the result file URL generator
Using FQDN of the worker's host name for the file-based protocol options XROOT and HTTP. The change allow running the application in the Kubernetes environment.
Configuration menu - View commit details
-
Copy full SHA for 1b6323c - Browse repository at this point
Copy the full SHA 1b6323cView commit details -
Fixed a bug in the result folder cleanup at the worker start up time
In the original implementation of the file-based result delivery protocol, workers were attempting to clean up unclaimed files left in workers' result folders regardless of the protocol option. This clearly was a mistake for the SSI protocol option where the folder wasn't required to exist. As a result of this, the application posts confusing warnings in the logging stream. This was fixed. Another problem that was fixed was related to the protocol options HTTP and XROOT where the application wouldn't abort right away if the required folder didn't exist during the folder cleanup attempt. Not having this folder available in this scenario means a configuration error or a problem with the infrastructure.
Configuration menu - View commit details
-
Copy full SHA for 0a51dad - Browse repository at this point
Copy the full SHA 0a51dadView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6dbcb49 - Browse repository at this point
Copy the full SHA 6dbcb49View commit details
There are no files selected for viewing