How to integrate with zipkin

[shuoli84]

Do you have a tutorial or doc to show how to integrate with zipkin? 💃

[jtolio]

Geez, that'd be a neat idea.

I'm going to leave this issue open until I write up good docs but to unblock you, your main method gets set up something like this:

package main

import (
  "net/http"

  "gopkg.in/spacemonkeygo/monkit.v2"
  "gopkg.in/spacemonkeygo/monkit.v2/environment"
  "gopkg.in/spacemonkeygo/monkit.v2/present"
  "gopkg.in/spacemonkeygo/monkit-zipkin.v2"
)

func main() {
  environment.Register(monkit.Default)
  go http.ListenAndServe("localhost:9000", present.HTTP(monkit.Default))
  collector, err := zipkin.NewScribeCollector("zipkin.whatever:9042")
  if err != nil { 
    panic(err)
  }
  zipkin.RegisterZipkin(monkit.Default, collector, zipkin.Options{})
}

Once you've done that, the only remaining thing is to make sure your HTTP handlers pull Zipkin Context info from the Request. That's easy to do with zipkin.ContextWrapper.

func HandleRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {
  defer mon.Task()(&ctx)(nil)

  ... whatever
}

func DoTheThing(ctx context.Context) (err error) {
  defer mon.Task()(&ctx)(&err)
  return http.Serve(listener, zipkin.ContextWrapper(zipkin.ContextHTTPHandlerFunc(HandleRequest)))  
}

Last, your outbound HTTP requests need to pass through Context info:

func MakeRequest(ctx context.Context) (err error) {
  defer mon.Task()(&ctx)(&err)
  req, err := http.NewRequest(...)
  if err != nil {
    return err
  }
  resp, err := zipkin.TraceRequest(ctx, http.DefaultClient, req)
  ...
}

Does that help?

[shuoli84]

Thanks!
Now golang 1.7 request has a method to get its context, and that is what I am doing, it should work too, right?

func HandleRequest(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    defer mon.Task()(&ctx)(nil)

  ... whatever
}

[shuoli84]

Hi, I have questions. I'm new to zipkin. So I may made some dump mistakes.

Here is what I did:

docker run -d -p 9411:9411 -p 9410:9410 openzipkin/zipkin

Then in my code

environment.Register(monkit.Default)
collector, err := zipkin.NewScribeCollector("0.0.0.0:9410"); checkErr(err)
zipkin.RegisterZipkin(monkit.Default, collector, zipkin.Options{})

I guess it should be able to connect to the collector.

Then I send some request to my service but nothing shows up in zipkin http://127.0.0.1:9411 (started from docker).

Any idea what I did wrong? I already setup the mokit trace as following:

Thanks in advance.

[jtolio]

Sorry to get back to you slowly:

First off, I wanted to apologize for my last example. This line is wrong:

  return http.Serve(listener, zipkin.ContextWrapper(zipkin.ContextHTTPHandlerFunc(HandleRequest)))  

It should really be:

  return http.Serve(listener, zipkin.ContextWrapper(zipkin.TraceHandler(zipkin.ContextHTTPHandlerFunc(HandleRequest))))  

The addition of the zipkin.TraceHandler method is key. Without it, you're right, your example using (*Request).Context is the same as what I said. But you do want to use zipkin.TraceHandler so that you can get remote trace information propagated from services that call your service.

We should definitely update our code so that it uses the new (*Request).Context stuff, that's neat.

As for why Zipkin isn't getting any traces, the default zipkin.Options struct won't send any traces unless they are opted-in using the zipkin.TraceHandler functionality (an incoming HTTP request has the appropriate zipkin X-B3-Span-Id headers and so forth). If you want all of your spans sent to Zipkin, you'll need to set different options in the zipkin.Options struct you're creating.

[jtolio]

I updated the docs for https://github.com/spacemonkeygo/monkit-zipkin/ to suggest zipkin.Options{Fraction: 1} by default

[shuoli84]

Hi jtolds, thanks for the reply.

I've modified my code to use zipkin.Options{Fraction: 1}. And also created one middleware like following

func TraceHandlerFor17Context(c http.Handler) http.Handler {
    return http.Handler(func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()
        trace, spanId := RequestFromHeader(r.Header).Trace()
        defer httpserver.FuncNamed(r.Method).RemoteTrace(&ctx, spanId, trace)(nil)
        s := monkit.SpanFromCtx(ctx)
        s.Annotate("http.uri", r.RequestURI)
        wrapped := &responseWriterObserver{w: w}
        c.ServeHTTP(wrapped, r)
        s.Annotate("http.responsecode", fmt.Sprint(wrapped.StatusCode()))
    })
}

The problem is nothing showed in zipkin, but I can see some logging in tcpdump.

13:04:56.988833 IP 127.0.0.1.58538 > 127.0.0.1.9410: Flags [.], ack 703, win 12737, options [nop,nop,TS val 3027082380 ecr 3027082380], length 0
E..4..@.@.............$.<...hq....1..(.....
.m...m..
13:04:56.988897 IP 127.0.0.1.58538 > 127.0.0.1.9410: Flags [P.], seq 6739:6743, ack 703, win 12737, options [nop,nop,TS val 3027082380 ecr 3027082380], length 4
E..8{.@.@.............$.<...hq....1..,.....
.m...m......
13:04:56.988911 IP 127.0.0.1.58538 > 127.0.0.1.9410: Flags [P.], seq 6743:6984, ack 703, win 12737, options [nop,nop,TS val 3027082380 ecr 3027082380], length 241
E..%..@.@.............$.<...hq....1........
.m...m..........Log...................zipkin.......CgABKZK+s+sVQyQLAAMAAAA5Z2l0bGFiLmV2ZXJwaG90by5jbi9mYWNlYXBpL2FwaS1zZXJ2ZXIvZGF0YWJhc2UuZGF0YXN0b3JlCgAEKZK+s+sVQyQPAAYMAAAAAgoAAQAFO1cJ2VWXCwACAAAAAmNzAAoAAQAFO1cJ2V02CwACAAAAAmNyAA8ACAwAAAAAAA==..
13:04:56.988915 IP 127.0.0.1.9410 > 127.0.0.1.58538: Flags [.], ack 6743, win 12548, options [nop,nop,TS val 3027082380 ecr 3027082380], length 0
E..4c.@.@...........$...hq..<.....1..(.....
.m...m..
13:04:56.988926 IP 127.0.0.1.9410 > 127.0.0.1.58538: Flags [.], ack 6984, win 12541, options [nop,nop,TS val 3027082380 ecr 3027082380], length 0
E..4.v@.@...........$...hq..<.....0..(.....
.m...m..
13:04:56.991235 IP 127.0.0.1.9410 > 127.0.0.1.58538: Flags [P.], seq 703:730, ack 6984, win 12541, options [nop,nop,TS val 3027082382 ecr 3027082380], length 27
E..O3#@.@...........$...hq..<.....0..C.....
.m...m..............Log............
13:04:56.991259 IP 127.0.0.1.58538 > 127.0.0.1.9410: Flags [.], ack 730, win 12736, options [nop,nop,TS val 3027082382 ecr 3027082382], length 0
E..4Y+@.@.............$.<...hq.3..1..(.....
.m...m..
13:04:56.991308 IP 127.0.0.1.58538 > 127.0.0.1.9410: Flags [P.], seq 6984:6988, ack 730, win 12736, options [nop,nop,TS val 3027082382 ecr 3027082382], length 4
E..8>   @.@.............$.<...hq.3..1..,.....
.m...m......
13:04:56.991318 IP 127.0.0.1.58538 > 127.0.0.1.9410: Flags [P.], seq 6988:7229, ack 730, win 12736, options [nop,nop,TS val 3027082382 ecr 3027082382], length 241
E..%..@.@.............$.<...hq.3..1........
.m...m..........Log...................zipkin.......CgABCLod6AHgl3ULAAMAAAA5Z2l0bGFiLmV2ZXJwaG90by5jbi9mYWNlYXBpL2FwaS1zZXJ2ZXIvZGF0YWJhc2UuZGF0YXN0b3JlCgAECLod6AHgl3UPAAYMAAAAAgoAAQAFO1cJ2U03CwACAAAAAmNzAAoAAQAFO1cJ2V1LCwACAAAAAmNyAA8ACAwAAAAAAA==..

What bugs me is I don't know how to trouble shoot the zipkin protocol yet. Any idea what I missed?

[alecholmez]

@shuoli84 I think the issue is the docker image. I don't think that the proper connection string to the docker container is 0.0.0.0:9410 but {docker_container_ip}:9410.

I'm trying to get zipkin setup in a docker-compose environment but I haven't had any luck

I actually got a thrift worker pool to start but still nothing in the UI

http-server_1   | 2017/01/12 03:34:56 GET       /       Docs    114.991µs
zipkin          | 2017-01-12 03:34:56.564  INFO 7 --- [thrift-worker-0] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Started.
http-server_1   | 2017/01/12 03:35:03 GET       /users  List All Users  519.468µs
zipkin          | 2017-01-12 03:35:13.627  INFO 7 --- [nio-9411-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring FrameworkServlet 'dispatcherServlet'
zipkin          | 2017-01-12 03:35:13.627  INFO 7 --- [nio-9411-exec-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization started
zipkin          | 2017-01-12 03:35:13.642  INFO 7 --- [nio-9411-exec-1] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization completed in 15 ms

Update

Looks like I'm getting a HTTP Request header error:

zipkin          | 2017-01-12 03:49:51.806  INFO 6 --- [nio-9411-exec-1] o.apache.coyote.http11.Http11Processor   : Error parsing HTTP request header
zipkin          |  Note: further occurrences of HTTP header parsing errors will be logged at DEBUG level.
zipkin          | 
zipkin          | java.lang.IllegalArgumentException: Invalid character found in method name. HTTP method names must be tokens
zipkin          |       at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:462) ~[tomcat-embed-core-8.5.6.jar!/:8.5.6]
zipkin          |       at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:667) ~[tomcat-embed-core-8.5.6.jar!/:8.5.6]
zipkin          |       at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
zipkin          |       at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
zipkin          |       at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
zipkin          |       at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
zipkin          |       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_112]
zipkin          |       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_112]
zipkin          |       at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-8.5.6.jar!/:8.5.6]
zipkin          |       at java.lang.Thread.run(Thread.java:745) [na:1.8.0_112]
zipkin          | 

Not sure why this isn't working

Update

I've been able to get by the errors but I still can't get anything into the zipkin-ui... I copied there example docker configuration but that still yielded nothing