How it Works

1. Dependency Injection
2. Bootstrap
3. Configurations 3.1. Additional Configurations 3.2. Spark Configurations 3.3. Hive support

Dependency Injection

Internally Nightfall uses Netflix Governator and Google Guava to provide the injection, which can only happen at the driver side of a Spark application. Due to serialization of functions is no possible at the time to use injection on a worker.

Nightfall enables the development stage of Guava, so all beans are eagerly created. We do this to avoid problems with lazy initialization with the injected beans.

The Nightfall jar distributted at maven central contains the org.objectweb.asm shaded to avoid conflicts with the one used by Spark, see add Nightfall to your project for more information.

To use Nightfall you just need to annotate your main class with one provider:

import com.elo7.nightfall.di.NightfallApplication;
import com.elo7.nightfall.di.providers.kafka.KafkaSimple;

@Nightfall(ExecutionMode.BATCH)
public class ExampleStreamApplication {

    public static void main(String[] args) {
        NightfallApplication.run(ExampleStreamApplication.class, args);
    }
}

Nightfall flow:

• Bootstrap the application;
• Look for all classes annotated with @Task;
• Added all classes annotated with @ModuleProvider as a Module;
• Creates the SparkSession;
• Run the Tasks. At this point a SparkSession can be injected.

Bootstrap

When an Nightfall application starts the run method creates the Injector using the NightfallBootStrapModule as the bootstrap module.

The NightfallBootStrapModule loads the properties from sources indicated at the runtime and bind it, so later injection can be performed.

By default Nightfall will scan all classes in the Nightfall package and in the main class package. Classes annotated with com.elo7.nightfall.di.Component and com.elo7.nightfall.di.Task are candidates for binding implementations.

Configurations

Nightfall supports three sources of configuration:

• Class path, which will look in the class path for the nightfall.properties file, this is the default method for Nightfall to load properties;
• Local file through the runtime flag -e file://file/path.properties;
• From Zookeeper through the runtime flag -e zookeeper://host:port,host2:port/file/path.properties.

The file is loaded from the source and all it's properties can be injected using Governator configuration mapping. Also note that Nightfall uses CompositeConfigurationProvider, which include all system propreties.

Example:

java -jar MyApplication.jar -e file://file/path.properties

The class NightfallConfigurations contains all configurations loaded from your configurations file, and it can be injected whatever class you need.

Additional Configurations

Nightfall also supports configurations throught command line, and these configurations override any found in the properties file:

Example:

java -jar MyApplication.jar -e file://file/path.properties -c "my.conf=foo" -c "my.another=bar"

Properties file content:

some=value
my.conf=default
my.another=some

The final properties would result in:

some=value
my.conf=foo
my.another=bar

Spark Configurations

Any configuration that contains the prefix spark. or hive. will be set in SparkConf, so if you have spark.app.name, Nightfall will set this property on SparkConf. See Spark Configuration for more information.

Hive Support

Hive supported is enabled during the Spark Session creation, as this is done by Nightfall you to need to set nightfall.spark.hive.enable=true.