Remove Spark Vector

[fschueler]

This removes the Spark Vector and the XORShift (since the vector was the only class depending on it).
The Kmeans is adjusted to use breeze DenseVectors.

[joroKr21]

Always happy to remove stuff, but it doesn't run with Flink or Spark (some weird compilation error).

[fschueler]

Hmm...I can't reprocude it with mvn clean package.

How/where did you get the error?

[joroKr21]

mvn clean package doesn't run the example tests due to excessive build time. You need to run mvn test -PIT -P[flink|spark] (IT stands for integration test).

[fschueler]

OK, I see it now. Will investigate!

[fschueler]

The tests pass now.

I would like to rewrite parts of Kmeans using breeze.numerics.pow() but using it makes compilation fail with:

Error:(25, 36) exception during macro expansion: 
scala.reflect.macros.TypecheckException: package is not an enclosing class
    at scala.reflect.macros.contexts.Typers$$anonfun$typecheck$2$$anonfun$apply$1.apply(Typers.scala:34)
    at scala.reflect.macros.contexts.Typers$$anonfun$typecheck$2$$anonfun$apply$1.apply(Typers.scala:28)
    at scala.reflect.macros.contexts.Typers$$anonfun$3.apply(Typers.scala:24)
    at scala.reflect.macros.contexts.Typers$$anonfun$3.apply(Typers.scala:24)
    at scala.reflect.macros.contexts.Typers$$anonfun$withContext$1$1.apply(Typers.scala:25)
    at scala.reflect.macros.contexts.Typers$$anonfun$withContext$1$1.apply(Typers.scala:25)
    at scala.reflect.macros.contexts.Typers$$anonfun$1.apply(Typers.scala:23)
    at scala.reflect.macros.contexts.Typers$$anonfun$1.apply(Typers.scala:23)
    at scala.reflect.macros.contexts.Typers$class.withContext$1(Typers.scala:25)
    at scala.reflect.macros.contexts.Typers$$anonfun$typecheck$2.apply(Typers.scala:28)
    at scala.reflect.macros.contexts.Typers$$anonfun$typecheck$2.apply(Typers.scala:28)
    at scala.reflect.internal.Trees$class.wrappingIntoTerm(Trees.scala:1716)
    at scala.reflect.internal.SymbolTable.wrappingIntoTerm(SymbolTable.scala:16)
    at scala.reflect.macros.contexts.Typers$class.withWrapping$1(Typers.scala:26)
    at scala.reflect.macros.contexts.Typers$class.typecheck(Typers.scala:28)
    at scala.reflect.macros.contexts.Context.typecheck(Context.scala:6)
    at scala.reflect.macros.contexts.Context.typecheck(Context.scala:6)
    at eu.stratosphere.emma.macros.BlackBoxUtil$class.typeCheck(BlackBoxUtil.scala:18)
    at eu.stratosphere.emma.macros.program.WorkflowMacros.typeCheck(WorkflowMacros.scala:11)
    at eu.stratosphere.emma.macros.ReflectUtil$syntax$TreeOps.typeChecked(ReflectUtil.scala:442)
    at eu.stratosphere.emma.macros.program.controlflow.ControlFlowNormalization$class.normalize(ControlFlowNormalization.scala:26)
    at eu.stratosphere.emma.macros.program.WorkflowMacros.normalize(WorkflowMacros.scala:11)
    at eu.stratosphere.emma.macros.program.WorkflowMacros.parallelize(WorkflowMacros.scala:27)
  val algorithm = emma.parallelize {
                                   ^

[aalexandrov]

Seems like something that might be broken due to #21. Might be good to ping @ParkL if you find the root of the issue so he can fix it while working on the semantic checks.

[joroKr21]

The problem is that we're still doing the typecheck . untypecheck thing in the end and it somehow messes up the normalized implicit arguments. There's a quick workaround, but ideally we should construct the Algorithm in the same fashion as with the other mk._ constructs.

[aalexandrov]

@fschueler Felix, can you rewrite the Algorithm assembly code @joroKr21 suggests?

[fschueler]

I will give it a try! Should I open a new issue and PR for that or reopen the old typecheck/untypecheck issue?

[aalexandrov]

You can include the commit as part of this PR. As this fix is required for the replacement of the vector implementation, I guess it makes sense to prepend it, so we have a commit history which transitions from consistent to consistent state.

[joroKr21]

The problem seems related to this PR, but we didn't manage to compile with 2.12-M3.

[fschueler]

Should we merge this without the changes in the Algorithm assembly? The tests pass without using breeze.numerics.pow() (doing "manual" multiplication)

[aalexandrov]

@fschueler yes, we can do that, but please open a separate issue for the Breeze problem that summarizes the findings from this PR.

[fschueler]

@aalexandrov done: #152

[joroKr21]

Merged via 76b22ee