Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: lsst/qserv
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 2023.7.1-rc3
Choose a base ref
...
head repository: lsst/qserv
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref

Commits on Jul 19, 2023

  1. Fixed a bug in qhttp response handler

    The older version of the code was throwing an exception when creating
    HTTP headers for file sizes exceeding signed 32-bit representation (~2 GB).
    iagaponenko committed Jul 19, 2023
    Copy the full SHA
    8c799cb View commit details
  2. Copy the full SHA
    790e379 View commit details
  3. Add support for the file-based result delivery to worker's CMSD config

    Added the corresponding configuration option to the entry-point CLI
    for ingecting a value of the result folder into the config file of
    of the worker CMSD. The configuration file has been extended as well
    to allow serving files from workers via the XROOTD file-based protocol.
    The SSI provider class has been extened to recognise result files
    at the local filesystems of workers as valid XROOTD resources.
    iagaponenko committed Jul 19, 2023
    Copy the full SHA
    ae9a111 View commit details
  4. Refactored and extended worker configuration service

    The service is now available to the client code via shared pointer
    that can be stored and used by classes as needed. The change reduces
    the number of parameters which are sent around the code. In the new
    version of the code only the shared pointer to the service is put
    to where the configuration parameters are consumed.
    
    Also added new parameters to support the file-based result delivery,
    including: a location of the results folder, the number of BOOST ASIO
    threads to run the QHTTP server at worker, and a selector for the desired
    results delivery protocol.
    iagaponenko committed Jul 19, 2023
    Copy the full SHA
    27ab504 View commit details
  5. Extended Protobuf definition to return file resource locations

    Also elminated deprecated protobuf schema and large result attributes
    iagaponenko committed Jul 19, 2023
    Copy the full SHA
    fab0082 View commit details
  6. Major refactoring and code cleanup in the worker query processing cla…

    …sses
    
    Split wbase::SendChannelShared into the base class and an implementation
    
    Did the minor refactoring in naming and using XROOTD/SSI streaming channels
    These change were required to prepare ground for introducing other
    "ChannelShared"-alike implementations of the result processing and delivery
    mechanisms.
    
    The refactoring aims at correcting dependencies in a code that creates
    and sets up worker tasks, so that task initialization was happening
    completelly within the implementation file of wbase/Task.cc rather
    than being spread between wbase/Task.cc and wcontrol/Foreman.cc.
    Altogether this is meant to improve the observability of the code and
    make it easier to maintain.
    
    Also, some minor code cleanup and removal of unused header includes
    and forward declarations was made. Thould reduce the compilation
    time of the relevant modules.
    
    Got rid of unused members and methods
    
    Classes wbase::ChannelShared (used to be wbase::SendChannelShared)
    and wbase::TransmitData had an obsolete mechanism for setting up
    result schema in the response messages to be sent to Czar. The current
    implementation no longer uses that API. The obsilete API was removed
    to avoid confusions as it wasn't obvious where the actual schema
    settings were made.
    
    Got rid of std:: in the iomplementation files with "using std".
    
    Guaranteed and enforced synchronization in private methods that require
    a lock to be held before calling the methods. In the new code, a cons
    reference to a lock is passed around contexts (methods) where such lock
    is required. This compiler-enforced technique has proven to work for large
    code bases where it would be hard to track a state of the lock by other methods.
    The only other alternative method would be to do the run-time inspection of
    the lock at the entrance of a method.
    iagaponenko committed Jul 19, 2023
    Copy the full SHA
    7e4dd53 View commit details
  7. Implemented writing and serving result files by workers

    The new result writer redirects result sets into files at workers.
    Only the last "summary" message with no rows is sent to Czar
    for each worker query (regardless if a number of tasks that were
    required to process the query).
    Also did the minor refactoring in the error messages
    
    Added HTTP server to Qserv worker for serving result files
    
    Also made minor fixes and improvements to basic tests of QHTTP
    made in the Replication/Ingest system code base.
    
    Using a configuration option to select the desired result delivery protocol
    
    Added support (is configured) for deleting unclaimed result files
    at the startup time of the worker service and after restarting the Czar.
    iagaponenko committed Jul 19, 2023
    Copy the full SHA
    526eaa1 View commit details
  8. Eliminated dead code in the result merger at Czar

    It was the trivial refactpring of the code meant to simplify the code
    and prepare it for the functional extention.
    iagaponenko committed Jul 19, 2023
    Copy the full SHA
    0376450 View commit details
  9. Modified Czar to allow pulling result files from workers

    The new version of the result merger at Czar would dynamically determine
    which protocol should be used for pulling results from workers: SSI stream,
    a file read via the XROOTD file protocol, or a file read via the HTTP
    protocol. In case of the last two options, the merger would also also
    utomatically tell a worker to delete the result file upon completion of
    the merge (including the unsuccessful ones).
    
    A choice of the delivery method is based on the optional
    fileds in the Protobuf messages received from workers.
    It's up to the workers to decide on what method to select.
    iagaponenko committed Jul 19, 2023
    Copy the full SHA
    4260547 View commit details
  10. Added a monitor for the results filesystem

    Also, extended the Web Dashboard to display the filesystem status
    and usage statistics.
    iagaponenko committed Jul 19, 2023
    Copy the full SHA
    77ffa3e View commit details

Commits on Jul 20, 2023

  1. Copy the full SHA
    fe21aa3 View commit details

Commits on Jul 21, 2023

  1. Refactored configuration service of Czar

    The service is now implemented as a singleton which is initialized
    and loaded with the configuration parameters at a time when Czar
    is being created. The rest of the Czar fetches values of the parameters
    from the singleton. The refactoring aims at reducing the number of parameters
    sent around in the code and to improve readability of the code.
    iagaponenko committed Jul 21, 2023
    Copy the full SHA
    bb80f0b View commit details
  2. Copy the full SHA
    38bac57 View commit details
  3. Enforce large result limit at workers when fetching result sets

    Note that the enforcement only applies to the file-based result
    delivery protocol as this mechanism makes no sense when results
    are streamed to Czar over the SSI protocol.
    
    The large result conditions are reported to Czar with the code
    util::ErrorCode::WORKER_RESULT_TOO_LARGE which would help Czar
    to identify this specific condition and properly report the one
    to a user.
    iagaponenko committed Jul 21, 2023
    Copy the full SHA
    53b951a View commit details
  4. Reinforced response processing in the result merger

    The new code verifies if the summary messages reported for the file-based
    results don't have any error contexts. Abort result merging should
    any such contexts be found.
    iagaponenko committed Jul 21, 2023
    Copy the full SHA
    77b1680 View commit details
  5. Copy the full SHA
    951c35d View commit details
  6. Fixed a candidate bug in the MySQL query cancellation

    Also, added a logging statement on failures to do so.
    iagaponenko committed Jul 21, 2023
    Copy the full SHA
    044b146 View commit details

Commits on Jul 22, 2023

  1. Copy the full SHA
    655206f View commit details

Commits on Aug 1, 2023

  1. Copy the full SHA
    7344288 View commit details
  2. Copy the full SHA
    c661f0f View commit details
  3. Migrated REST API of the Replication system

    The new implementation of the Qserv query search service no longer
    uses the full-text indexes on the query text. The service now relies
    on the simpler search methods LIKE and REGEXP.
    
    The version number of the API is now 23.
    iagaponenko committed Aug 1, 2023
    Copy the full SHA
    f8e1c8c View commit details
  4. Web Dashboard: migrated to follow changes in the REST API

    Changes affect the search for for the past queries.
    iagaponenko committed Aug 1, 2023
    Copy the full SHA
    4221c20 View commit details
  5. Copy the full SHA
    94847e6 View commit details

Commits on Aug 10, 2023

  1. The minor extension made to the MySQL connector

    Also added the utility class to handle simple query cases.
    iagaponenko committed Aug 10, 2023
    Copy the full SHA
    42b7db6 View commit details

Commits on Aug 17, 2023

  1. Moderate refactoring and code consolidation in the worker requests

    Simplified error reporting. Reduced code duplication.
    iagaponenko committed Aug 17, 2023
    Copy the full SHA
    bc0ca76 View commit details
  2. Copy the full SHA
    20293e6 View commit details
  3. Copy the full SHA
    73f72f3 View commit details
  4. Copy the full SHA
    62a32be View commit details
  5. Extended Replication system's database API

    Added a function for getting info on the ongoing database activities
    as reported by the query SHOW [FULL] PROCESSLIST.
    iagaponenko committed Aug 17, 2023
    Copy the full SHA
    99c7e20 View commit details
  6. Extended Controller's REST API to report database activities

    Added the REST services to report database activities at Czar and
    Qserv workers. Incremented the version number of the REST API.
    iagaponenko committed Aug 17, 2023
    Copy the full SHA
    951df6e View commit details
  7. Copy the full SHA
    098cf37 View commit details
  8. Copy the full SHA
    3f550dc View commit details
  9. Capture MySQL thread identifiers for worker queries

    The identifiers are captured for the worker task monitoring
    purposes. They are used by the Web Dashboard for associating task
    activities with the corresponding MySQL queries issued by the tasks.
    The task monitoring page of the Dashboard has been extended to display
    a value of the identifier (if it was set for the task).
    iagaponenko committed Aug 17, 2023
    Copy the full SHA
    a502f15 View commit details
  10. Refined states of the worker tasks

    Introduced an additional state STARTED to differentiate between
    two events - when a task was started by a scheduler from the actual
    start time of the corresponding MySQL query. The change is meant
    to improve the monitoring of the worker tasks.
    The corresponding monitoring page of the Web Dashboard was modified
    as well.
    iagaponenko committed Aug 17, 2023
    Copy the full SHA
    e22ca72 View commit details
  11. Copy the full SHA
    cefbb18 View commit details
  12. Copy the full SHA
    4971fc5 View commit details

Commits on Aug 24, 2023

  1. Fixed a synchronization bug in the helper class

    Minor refactoring in the class.
    iagaponenko committed Aug 24, 2023
    Copy the full SHA
    b3d4bcc View commit details

Commits on Aug 28, 2023

  1. Copy the full SHA
    9aaf64f View commit details

Commits on Sep 7, 2023

  1. Copy the full SHA
    88eb123 View commit details
  2. Copy the full SHA
    80fa2b4 View commit details
  3. Copy the full SHA
    cf19796 View commit details
  4. Copy the full SHA
    5303ba9 View commit details
  5. Copy the full SHA
    80f6786 View commit details
  6. Copy the full SHA
    33f04ba View commit details

Commits on Sep 8, 2023

  1. Copy the full SHA
    3f76000 View commit details
  2. Merge pull request #808 from lsst/u/fritzm/readme-main-fix

    Fix master->main problems in README.md
    fritzm authored Sep 8, 2023
    Copy the full SHA
    db15f28 View commit details

Commits on Sep 14, 2023

  1. Moved the function for finding host's FQDN(s) into util/

    Extended the contract and the implementation of the function to
    allow a choice to return a single result or all findings.
    Migrated the only client of that function that existed before
    the move.
    iagaponenko committed Sep 14, 2023
    Copy the full SHA
    110f6ee View commit details
  2. Fixed a bug in the result file URL generator

    Using FQDN of the worker's host name for the file-based protocol
    options XROOT and HTTP. The change allow running the application
    in the Kubernetes environment.
    iagaponenko committed Sep 14, 2023
    Copy the full SHA
    1b6323c View commit details
  3. Fixed a bug in the result folder cleanup at the worker start up time

    In the original implementation of the file-based result delivery protocol,
    workers were attempting to clean up unclaimed files left in workers'
    result folders regardless of the protocol option. This clearly was
    a mistake for the SSI protocol option where the folder wasn't
    required to exist. As a result of this, the application posts
    confusing warnings in the logging stream. This was fixed.
    
    Another problem that was fixed was related to the protocol options HTTP
    and XROOT where the application wouldn't abort right away if the required
    folder didn't exist during the folder cleanup attempt. Not having this
    folder available in this scenario means a configuration error or
    a problem with the infrastructure.
    iagaponenko committed Sep 14, 2023
    Copy the full SHA
    0a51dad View commit details
  4. Copy the full SHA
    6dbcb49 View commit details
Showing 1,076 changed files with 57,039 additions and 33,410 deletions.
184 changes: 108 additions & 76 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -3,9 +3,6 @@
name: "CI"
on:
push:
pull_request:
branches:
- main
workflow_dispatch:

jobs:
@@ -27,7 +24,7 @@ jobs:
# could change from job to job, breaking the running workflow in a difficult
# to understand way.
name: Save image names
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
outputs:
build-image: ${{ steps.image-names.outputs.build-image }}
user-build-image: ${{ steps.image-names.outputs.user-build-image }}
@@ -37,7 +34,7 @@ jobs:

steps:
- name: Install python
uses: actions/setup-python@v3
uses: actions/setup-python@v5
with:
python-version: '3.x'

@@ -47,7 +44,7 @@ jobs:
pip install click pyyaml requests
- name: Checkout code
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
fetch-depth: 0 # 0 is "all history and branch tags"

@@ -65,7 +62,7 @@ jobs:
update-base-images:
name: Update base images
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
needs: image-names
outputs:
build-rebuilt: ${{ steps.rebuild.outputs.build-rebuilt }}
@@ -74,7 +71,7 @@ jobs:
steps:

- name: Install python
uses: actions/setup-python@v3
uses: actions/setup-python@v5
with:
python-version: '3.x'

@@ -84,12 +81,12 @@ jobs:
pip install click pyyaml requests
- name: Checkout code
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
fetch-depth: 0 # 0 is "all history and branch tags"

- name: Login to DockerHub
uses: docker/login-action@v2
uses: docker/login-action@v3
with:
username: ${{ secrets.FM_DOCKERHUB_TOKEN_USER }}
password: ${{ secrets.FM_DOCKERHUB_TOKEN }}
@@ -125,14 +122,14 @@ jobs:

update-mariadb-image:
name: Update MariaDB image
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
needs: image-names
outputs:
rebuilt: ${{ steps.rebuild.outputs.rebuilt }}
steps:

- name: Install python
uses: actions/setup-python@v3
uses: actions/setup-python@v5
with:
python-version: '3.x'

@@ -142,12 +139,12 @@ jobs:
pip install click pyyaml requests
- name: Checkout code
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
fetch-depth: 0 # 0 is "all history and branch tags"

- name: Login to DockerHub
uses: docker/login-action@v2
uses: docker/login-action@v3
with:
username: ${{ secrets.FM_DOCKERHUB_TOKEN_USER }}
password: ${{ secrets.FM_DOCKERHUB_TOKEN }}
@@ -166,65 +163,14 @@ jobs:
QSERV_DH_USER: ${{ secrets.FM_DOCKERHUB_TOKEN_USER }}
QSERV_DH_TOKEN: ${{ secrets.FM_DOCKERHUB_TOKEN }}

documentation:
name: Documentation
runs-on: ubuntu-20.04
needs: [image-names, update-base-images]
steps:

- name: Install python
uses: actions/setup-python@v3
with:
python-version: '3.x'

- name: Install click
run: |
python -m pip install --upgrade pip
pip install click pyyaml requests
- name: Checkout code
uses: actions/checkout@v3
with:
submodules: true
fetch-depth: 0 # 0 is "all history and branch tags"

- name: Check for changes
uses: dorny/paths-filter@v2
id: filter
with:
filters: |
docs:
- '**'
- name: Prepare user build image
if: needs.update-base-images.outputs.build-rebuilt == 'True' || steps.filter.outputs.docs == 'true'
run: |
./admin/local/cli/qserv --log-level DEBUG build-user-build-image \
--group docker_outer \
--build-image ${{ needs.image-names.outputs.build-image }} \
--user-build-image ${{ needs.image-names.outputs.user-build-image }}
- name: Build and publish
if: needs.update-base-images.outputs.build-rebuilt == 'True' || steps.filter.outputs.docs == 'true'
run: |
./admin/local/cli/qserv build-docs --cmake --linkcheck --upload \
--user-build-image ${{ needs.image-names.outputs.user-build-image }}
env:
QSERV_LTD_USERNAME: ${{ secrets.LTD_USERNAME }}
QSERV_LTD_PASSWORD: ${{ secrets.LTD_PASSWORD }}
QSERV_GH_EVENT_NAME: ${{ github.event_name }}
QSERV_GH_HEAD_REF: ${{ github.head_ref }}
QSERV_GH_REF: ${{ github.ref }}


update-run-image:
name: Update Qserv image
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
needs: [image-names, update-base-images]
steps:

- name: Install python
uses: actions/setup-python@v3
uses: actions/setup-python@v5
with:
python-version: '3.x'

@@ -234,7 +180,7 @@ jobs:
pip install click pyyaml requests
- name: Checkout code
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
fetch-depth: 0 # 0 is "all history and branch tags"

@@ -243,7 +189,7 @@ jobs:
git submodule update --init
- name: Login to DockerHub
uses: docker/login-action@v2
uses: docker/login-action@v3
with:
username: ${{ secrets.FM_DOCKERHUB_TOKEN_USER }}
password: ${{ secrets.FM_DOCKERHUB_TOKEN }}
@@ -271,27 +217,32 @@ jobs:

compose-integration-tests:
name: Integration tests (compose)
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04
needs: [image-names, update-mariadb-image, update-run-image]
steps:

- name: Install python
uses: actions/setup-python@v3
uses: actions/setup-python@v5
with:
python-version: '3.x'

- name: Install docker-compose
uses: KengoTODA/actions-setup-docker-compose@v1
with:
version: '1.29.2'

- name: Install click
run: |
python -m pip install --upgrade pip
pip install click pyyaml requests
- name: Checkout code
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
fetch-depth: 0 # 0 is "all history and branch tags"

- name: Login to DockerHub
uses: docker/login-action@v2
uses: docker/login-action@v3
with:
username: ${{ secrets.FM_DOCKERHUB_TOKEN_USER }}
password: ${{ secrets.FM_DOCKERHUB_TOKEN }}
@@ -302,13 +253,94 @@ jobs:
--qserv-image ${{ needs.image-names.outputs.qserv-image }} \
--mariadb-image ${{ needs.image-names.outputs.mariadb-image }}
- name: Check Qserv containers
run: sleep 180 && docker ps -a

- name: Run integration tests
run: |
./admin/local/cli/qserv --log-level DEBUG itest \
--wait 180 \
--qserv-image ${{ needs.image-names.outputs.qserv-image }} \
--mariadb-image ${{ needs.image-names.outputs.mariadb-image }}
- name: Run integration tests of the HTTP frontend
run: |
./admin/local/cli/qserv --log-level DEBUG itest-http --reload --load-http \
--qserv-image ${{ needs.image-names.outputs.qserv-image }} \
--mariadb-image ${{ needs.image-names.outputs.mariadb-image }}
- name: Run integration tests of ingesting user tables via the HTTP frontend
run: |
./admin/local/cli/qserv --log-level DEBUG itest-http-ingest \
--qserv-image ${{ needs.image-names.outputs.qserv-image }}
- name: Check Qserv containers
if: always()
run: docker ps -a

- name: Replication Controller Log
if: always()
run: docker logs ${USER}-repl-controller-1

- name: Replication Registry Log
if: always()
run: docker logs ${USER}-repl-registry-1

- name: Replication Database Log
if: always()
run: docker logs ${USER}-repl-mariadb-1

- name: Replication Worker 0 Log
if: always()
run: docker logs ${USER}-repl-worker-0-1

- name: Replication Worker 1 Log
if: always()
run: docker logs ${USER}-repl-worker-1-1

- name: Czar MySQL Proxy Log
if: always()
run: docker logs ${USER}-czar-proxy-1

- name: Czar HTTP Frontend Log
if: always()
run: docker logs ${USER}-czar-http-1

- name: Czar CMSD Log
if: always()
run: docker logs ${USER}-czar-cmsd-1

- name: Czar XROOTD Log
if: always()
run: docker logs ${USER}-czar-xrootd-1

- name: Czar MariaDB Log
if: always()
run: docker logs ${USER}-czar-mariadb-1

- name: Qzerv Worker 0 CMSD Log
if: always()
run: docker logs ${USER}-worker-cmsd-0-1

- name: Qzerv Worker 0 XROOTD Log
if: always()
run: docker logs ${USER}-worker-xrootd-0-1

- name: Qzerv Worker 0 MariaDB Log
if: always()
run: docker logs ${USER}-worker-mariadb-0-1

- name: Qzerv Worker 1 CMSD Log
if: always()
run: docker logs ${USER}-worker-cmsd-1-1

- name: Qzerv Worker 1 XROOTD Log
if: always()
run: docker logs ${USER}-worker-xrootd-1-1

- name: Qzerv Worker 1 MariaDB Log
if: always()
run: docker logs ${USER}-worker-mariadb-1-1

- name: Remove integration test volumes
run: |
./admin/local/cli/qserv --log-level DEBUG itest-rm
@@ -320,8 +352,8 @@ jobs:
notify-on-fail:
name: Notify Slack if fail on main
runs-on: ubuntu-20.04
needs: [documentation, compose-integration-tests]
runs-on: ubuntu-22.04
needs: [compose-integration-tests]
if: github.ref == 'refs/heads/main' && failure()
steps:
- name: Notify
Loading