Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/logisland 532 #553

Merged
merged 43 commits into from
Apr 9, 2020
Merged

Feature/logisland 532 #553

merged 43 commits into from
Apr 9, 2020

Commits on Feb 4, 2020

  1. Fix url for jcenter.bintray.com repository: from http to https

    @mathieu-rossignol
    mathieu-rossignol committed Feb 4, 2020
    Configuration menu
    Copy the full SHA
    1bc36df View commit details
    Browse the repository at this point in the history

Commits on Feb 5, 2020

  1. Added databricks option to support databricks run: load config from D… 

    …BFS, using the DB spark context
    @mathieu-rossignol
    mathieu-rossignol committed Feb 5, 2020
    Configuration menu
    Copy the full SHA
    919c337 View commit details
    Browse the repository at this point in the history

  2. Introducing SparkConfigReader class to be packaged and loaded only in… 

    … databricks environment (no more useless spark dependency when outside of databricks)
    @mathieu-rossignol
    mathieu-rossignol committed Feb 5, 2020
    Configuration menu
    Copy the full SHA
    587814e View commit details
    Browse the repository at this point in the history

Commits on Feb 10, 2020

  1. Introducing logisland-engine-spark_2_4plus_kafka_2_4plus

    @mathieu-rossignol
    mathieu-rossignol committed Feb 10, 2020
    Configuration menu
    Copy the full SHA
    25fddb0 View commit details
    Browse the repository at this point in the history

  2. Attempt to add -checkpointLocation option

    @mathieu-rossignol
    mathieu-rossignol committed Feb 10, 2020
    Configuration menu
    Copy the full SHA
    4a36bdb View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2020

  1. Remove building of spark2.4_kafka2.4 engine for the moment

    @mathieu-rossignol
    mathieu-rossignol committed Feb 11, 2020
    Configuration menu
    Copy the full SHA
    f44321a View commit details
    Browse the repository at this point in the history

  2. Added spark 1.4 support in 2.x engine. Also added azure eventhubs spa… 

    …rk streaming dependencies
    @mathieu-rossignol
    mathieu-rossignol committed Feb 11, 2020
    Configuration menu
    Copy the full SHA
    bfdf3ec View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2020

  1. Saving intermediate step towards new service to support azure event hubs

    @mathieu-rossignol
    mathieu-rossignol committed Feb 13, 2020
    Configuration menu
    Copy the full SHA
    bee69e0 View commit details
    Browse the repository at this point in the history

  2. Eventhbubs service: all config properties read

    @mathieu-rossignol
    mathieu-rossignol committed Feb 13, 2020
    Configuration menu
    Copy the full SHA
    da0ce2e View commit details
    Browse the repository at this point in the history

  3. First attempt to have azure event hubs service running, will have to … 

    …test now...
    @mathieu-rossignol
    mathieu-rossignol committed Feb 13, 2020
    Configuration menu
    Copy the full SHA
    1075c46 View commit details
    Browse the repository at this point in the history

  4. Fixed spark 2.4 module pom mqtt dependency

    @mathieu-rossignol
    mathieu-rossignol committed Feb 13, 2020
    Configuration menu
    Copy the full SHA
    ab6a254 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2020

  1. Using checkpoint location in azure structured stream service. Also fi… 

    …xed a bug where read position configiguration property cannot be anything else than string
    @mathieu-rossignol
    mathieu-rossignol committed Feb 14, 2020
    Configuration menu
    Copy the full SHA
    3491194 View commit details
    Browse the repository at this point in the history

  2. Support spark 2.4 engine in logisland.sh script

    @mathieu-rossignol
    mathieu-rossignol committed Feb 14, 2020
    Configuration menu
    Copy the full SHA
    65d2f46 View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2020

  1. Update version of spark-sql-streaming-mqtt_2.11 dependency

    @mathieu-rossignol
    mathieu-rossignol committed Feb 17, 2020
    Configuration menu
    Copy the full SHA
    dc360dd View commit details
    Browse the repository at this point in the history

  2. Removed useless parenthesis

    @mathieu-rossignol
    mathieu-rossignol committed Feb 17, 2020
    Configuration menu
    Copy the full SHA
    086aeef View commit details
    Browse the repository at this point in the history

  3. Added spark 2 common jar to logisland doc dependency

    @mathieu-rossignol
    mathieu-rossignol committed Feb 17, 2020
    Configuration menu
    Copy the full SHA
    7ffb978 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2020

  1. Use right scala version

    @mathieu-rossignol
    mathieu-rossignol committed Feb 18, 2020
    Configuration menu
    Copy the full SHA
    280e34d View commit details
    Browse the repository at this point in the history

  2. Explicitly adding bahir-common dependency in spark2.4 engine, as a wo… 

    …rkaround to class not found for org.apache.bahir.utils.Logging when using AzureEventHubsStructuredStreamProviderService
    @mathieu-rossignol
    mathieu-rossignol committed Feb 18, 2020
    Configuration menu
    Copy the full SHA
    8d0ea78 View commit details
    Browse the repository at this point in the history

Commits on Feb 19, 2020

  1. Set checkpointlocation in structured stream

    @mathieu-rossignol
    mathieu-rossignol committed Feb 19, 2020
    Configuration menu
    Copy the full SHA
    f59dabc View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2020

  1. Added spark deploy mode property to support sparl standlone cluster d… 

    …eploy mode
    @mathieu-rossignol
    mathieu-rossignol committed Feb 21, 2020
    Configuration menu
    Copy the full SHA
    bbf270e View commit details
    Browse the repository at this point in the history

  2. Introudcing support of spark standalone cluster in logisland.sh

    @mathieu-rossignol
    mathieu-rossignol committed Feb 21, 2020
    Configuration menu
    Copy the full SHA
    2a719c5 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2020

  1. Fix for ClassNotFoundException in databricks environment on executor:… 

    … the plugin registry is not filled as the current thread context classloader is a repl.ExecutorClassLoader, so class of plugins cannot be loaded
    @mathieu-rossignol
    mathieu-rossignol committed Feb 26, 2020
    Configuration menu
    Copy the full SHA
    6bd742b View commit details
    Browse the repository at this point in the history

  2. Need spark 2.4.0 in spark 2.4 engine to align with latest supported v… 

    …ersion of azure event hub spark structured stream source/sink
    @mathieu-rossignol
    mathieu-rossignol committed Feb 26, 2020
    Configuration menu
    Copy the full SHA
    337f895 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2020

  1. Use offset as record key when reading from azure event hub. Also remo… 

    …ving some debug traces and adding some info logs
    @mathieu-rossignol
    mathieu-rossignol committed Feb 27, 2020
    Configuration menu
    Copy the full SHA
    7f5c0f7 View commit details
    Browse the repository at this point in the history

  2. Remove not finished and useless engine for spark 2.4+ and kafka 2.4+

    @mathieu-rossignol
    mathieu-rossignol committed Feb 27, 2020
    Configuration menu
    Copy the full SHA
    2e1de7d View commit details
    Browse the repository at this point in the history

  3. Update user doc after full build

    @mathieu-rossignol
    mathieu-rossignol committed Feb 27, 2020
    Configuration menu
    Copy the full SHA
    94732be View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2020

  1. Added experimental logisland.sh feature that allows minimal launch on… 

    … spark standalone cluster in client mode, not tested cluster mode. Also to be improved
    @mathieu-rossignol
    mathieu-rossignol committed Feb 28, 2020
    Configuration menu
    Copy the full SHA
    b4d1f77 View commit details
    Browse the repository at this point in the history

  2. Added OpenDistro Elasticsearch integration test. For the moment, only… 

    … user password without https is supported and validated
    @mathieu-rossignol
    mathieu-rossignol committed Feb 28, 2020
    Configuration menu
    Copy the full SHA
    af2e8b7 View commit details
    Browse the repository at this point in the history

  3. Centralized opendistro user name and password in integration tests

    @mathieu-rossignol
    mathieu-rossignol committed Feb 28, 2020
    Configuration menu
    Copy the full SHA
    bd0636a View commit details
    Browse the repository at this point in the history

  4. update test containers version in ES7 integration test

    @mathieu-rossignol
    mathieu-rossignol committed Feb 28, 2020
    Configuration menu
    Copy the full SHA
    85b1c6e View commit details
    Browse the repository at this point in the history

  5. No more need of mock service in ES 6 and 7 integration test: use real… 

    … client connection created by the real service himself
    @mathieu-rossignol
    mathieu-rossignol committed Feb 28, 2020
    Configuration menu
    Copy the full SHA
    e5cec1c View commit details
    Browse the repository at this point in the history

  6. Support for HTTPS opendistro elasticsearch

    @mathieu-rossignol
    mathieu-rossignol committed Feb 28, 2020
    Configuration menu
    Copy the full SHA
    2607fac View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2020

  1. Added comment in ES7 service regarding SSL

    @mathieu-rossignol
    mathieu-rossignol committed Mar 10, 2020
    Configuration menu
    Copy the full SHA
    e66b83a View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2020

  1. Added support for avro serializer/deserializer in structured streams … 

    …by supporting input/output avro schema configuration properties
    @mathieu-rossignol
    mathieu-rossignol committed Mar 11, 2020
    Configuration menu
    Copy the full SHA
    0b36c05 View commit details
    Browse the repository at this point in the history

  2. From avro 1.8.2 to avro 1.9.2 to have avro schema serializable and th… 

    …us now usable in structured streams
    @mathieu-rossignol
    mathieu-rossignol committed Mar 11, 2020
    Configuration menu
    Copy the full SHA
    557cbec View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2020

  1. Forgotten close when loading maxmind db from URI. Alos upgraded versi… 

    …ons of iptogeo service and processor dependencies (jackson/hadoop)
    @mathieu-rossignol
    mathieu-rossignol committed Mar 12, 2020
    Configuration menu
    Copy the full SHA
    84cb63a View commit details
    Browse the repository at this point in the history

  2. Changed log messages when loading maxmind DB from URI

    @mathieu-rossignol
    mathieu-rossignol committed Mar 12, 2020
    Configuration menu
    Copy the full SHA
    dd876d0 View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2020

  1. Rollback park-sql-streaming-mqtt_2.11 version to not have conflict wi… 

    …th bahir underlying dependency that uses scala 2.12 when using 2.4.0 version (don't know why)
    @mathieu-rossignol
    mathieu-rossignol committed Mar 13, 2020
    Configuration menu
    Copy the full SHA
    4343104 View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2020

  1. Fixed Record JSON deserialization problem after jackson version bump.… 

    … Also attempting to align bumped jackson version everywhere where possible and also use the jackson.version property
    @mathieu-rossignol
    mathieu-rossignol committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    0e5a5fa View commit details
    Browse the repository at this point in the history

  2. Changed http://jcenter.bintray.com into https://jcenter.bintray.com i… 

    …n ivy.xml
    @mathieu-rossignol
    mathieu-rossignol committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    c531acd View commit details
    Browse the repository at this point in the history

  3. Added https://repo1.maven.org/maven2 repo to ivy.xml

    @mathieu-rossignol
    mathieu-rossignol committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    2159598 View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2020

  1. Fix logisland.sh script using wrong spark engine: when spark 2.3.3 it… 

    … was using 2.4 engine, this made fail the structured stream travis integration test
    @mathieu-rossignol
    mathieu-rossignol committed Mar 17, 2020
    Configuration menu
    Copy the full SHA
    15ffd64 View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2020

  1. Try to fix 20 minutes timeout when running travis integration tests

    @mathieu-rossignol
    mathieu-rossignol committed Mar 18, 2020
    Configuration menu
    Copy the full SHA
    aaae6be View commit details
    Browse the repository at this point in the history