Unable to connect to history server hosted on Microsoft Azure

[NileshGule]

I am unable to add the history server link which is hosted on Microsoft Azure HDInsight cloud. I get the error

unexpected HTTP status: 401 Credentials missing or Auth is not Basic

Is there any way to provide the credentials?

[roboxue]

This is not an Application specific behavior either. (Simply no code is going to return a 401 here in this repo) It's likely being your Azure settings. I know some cloud service forbid certain ports by default, like AWS blocks almost every any port on EMR nodes.

You can provide a -p option to use an open port if you know it, otherwise contact your system admin's for this.

[NileshGule]

I have a Azure HDInsight cluster with public key access. I can connect to Ambari & other management UI as well as Spark history server using a browser & supplying the authentication parameters. How can I do the same when I want to add the link to History server using SparkLint UI?

[roboxue]

@NileshGule try modify these lines to add appropriate headers to see if this solves your problem. (I assume Azure uses header in requests to authenticate. (I can't find any doc online yet, and I'm not an Azure user to find out myself. You can probably look at the request using browser developer tool to see how the authentication was done when accessing HistoryServerUI.)

sparklint/src/main/scala/com/groupon/sparklint/events/HistoryServerApi.scala

Lines 84 to 86 in ce3e570

	val request = Request(uri = (apiUri / "applications")
	.withQueryParam("status", "completed")
	.withQueryParam("minDate", new SimpleDateFormat("yyyy-MM-dd").format(minDate)))