CompileError clang/cython

[rgdagir]

Hi!

I am trying to run the gryphon dashboards with the Gryphon Data Service (GDS). I have set up redis and RabbitMQ properly, but I keep on running into a CompileError when I call foreman start to start the GDS. Has anyone run into the same problem? Any guidance on how I should proceed? The TRADES and ORDERBOOK parts work fine, as shown in the error message below, but PRODUCER always crashes:

$ foreman start
11:43:09 PRODUCER.1           | started with pid 18413
11:43:09 ORDERBOOK_CONSUMER.1 | started with pid 18414
11:43:09 TRADES_CONSUMER.1    | started with pid 18415
11:43:09 VOLUMES_CONSUMER.1   | started with pid 18416
11:43:10 VOLUMES_CONSUMER.1   | INFO:raven.base.Client:Raven is not configured (logging is disabled). Please see the documentation for more information.
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:raven.base.Client:Raven is not configured (logging is disabled). Please see the documentation for more information.
11:43:10 VOLUMES_CONSUMER.1   | INFO:emerald-havoc-consumer:Connecting to amqp://guest:guest@localhost:5672
11:43:10 VOLUMES_CONSUMER.1   | INFO:pika.adapters.base_connection:Connecting to ::1:5672
11:43:10 VOLUMES_CONSUMER.1   | INFO:emerald-havoc-consumer:Connection opened
11:43:10 VOLUMES_CONSUMER.1   | INFO:emerald-havoc-consumer:Adding connection close callback
11:43:10 VOLUMES_CONSUMER.1   | INFO:emerald-havoc-consumer:Channel opened
11:43:10 VOLUMES_CONSUMER.1   | INFO:emerald-havoc-consumer:Adding channel close callback
11:43:10 VOLUMES_CONSUMER.1   | INFO:emerald-havoc-consumer:Declaring exchange trades_exchange
11:43:10 VOLUMES_CONSUMER.1   | INFO:emerald-havoc-consumer:Exchange declared
11:43:10 VOLUMES_CONSUMER.1   | INFO:emerald-havoc-consumer:Declaring queue exchange_volume_durable
11:43:10 VOLUMES_CONSUMER.1   | INFO:emerald-havoc-consumer:Binding trades_exchange to exchange_volume_durable with *.exchange_volume.tinker
11:43:10 VOLUMES_CONSUMER.1   | INFO:emerald-havoc-consumer:Queue bound
11:43:10 VOLUMES_CONSUMER.1   | INFO:emerald-havoc-consumer:Issuing consumer related RPC commands
11:43:10 VOLUMES_CONSUMER.1   | INFO:emerald-havoc-consumer:Adding consumer cancellation callback
11:43:10 TRADES_CONSUMER.1    | INFO:raven.base.Client:Raven is not configured (logging is disabled). Please see the documentation for more information.
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:emerald-havoc-consumer:Connecting to amqp://guest:guest@localhost:5672
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:pika.adapters.base_connection:Connecting to ::1:5672
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:emerald-havoc-consumer:Connection opened
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:emerald-havoc-consumer:Adding connection close callback
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:emerald-havoc-consumer:Channel opened
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:emerald-havoc-consumer:Adding channel close callback
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:emerald-havoc-consumer:Declaring exchange trades_exchange
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:emerald-havoc-consumer:Exchange declared
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:emerald-havoc-consumer:Declaring queue orderbook_durable
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:emerald-havoc-consumer:Binding trades_exchange to orderbook_durable with *.orderbook.tinker
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:emerald-havoc-consumer:Queue bound
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:emerald-havoc-consumer:Issuing consumer related RPC commands
11:43:10 ORDERBOOK_CONSUMER.1 | INFO:emerald-havoc-consumer:Adding consumer cancellation callback
11:43:10 TRADES_CONSUMER.1    | INFO:emerald-havoc-consumer:Connecting to amqp://guest:guest@localhost:5672
11:43:10 TRADES_CONSUMER.1    | INFO:pika.adapters.base_connection:Connecting to ::1:5672
11:43:10 TRADES_CONSUMER.1    | INFO:emerald-havoc-consumer:Connection opened
11:43:10 TRADES_CONSUMER.1    | INFO:emerald-havoc-consumer:Adding connection close callback
11:43:10 TRADES_CONSUMER.1    | INFO:emerald-havoc-consumer:Channel opened
11:43:10 TRADES_CONSUMER.1    | INFO:emerald-havoc-consumer:Adding channel close callback
11:43:10 TRADES_CONSUMER.1    | INFO:emerald-havoc-consumer:Declaring exchange trades_exchange
11:43:10 TRADES_CONSUMER.1    | INFO:emerald-havoc-consumer:Exchange declared
11:43:10 TRADES_CONSUMER.1    | INFO:emerald-havoc-consumer:Declaring queue trades_durable
11:43:10 TRADES_CONSUMER.1    | INFO:emerald-havoc-consumer:Binding trades_exchange to trades_durable with *.trades.tinker
11:43:10 TRADES_CONSUMER.1    | INFO:emerald-havoc-consumer:Queue bound
11:43:10 TRADES_CONSUMER.1    | INFO:emerald-havoc-consumer:Issuing consumer related RPC commands
11:43:10 TRADES_CONSUMER.1    | INFO:emerald-havoc-consumer:Adding consumer cancellation callback
11:43:10 PRODUCER.1           | /Users/raulgallodagir/.pyxbld/temp.macosx-10.14-x86_64-2.7/pyrex/gryphon/lib/exchange/base.c:17:2: error: unterminated conditional directive
11:43:10 PRODUCER.1           | #ifndef Py_PYTHON_H
11:43:10 PRODUCER.1           |  ^
11:43:10 PRODUCER.1           | /Users/raulgallodagir/.pyxbld/temp.macosx-10.14-x86_64-2.7/pyrex/gryphon/lib/exchange/base.c:14490:102: error: expected ';' after top level declarator
11:43:10 PRODUCER.1           |   {&__pyx_n_s_price_quote_req, __pyx_k_price_quote_req, sizeof(__pyx_k_price_quote_req), 0, 0, 1, 1}}
11:43:10 PRODUCER.1           |                                                                                                      ^
11:43:10 PRODUCER.1           |                                                                                                      ;
11:43:10 PRODUCER.1           | 2 errors generated.
11:43:10 PRODUCER.1           | Traceback (most recent call last):
11:43:10 PRODUCER.1           |   File "runt.py", line 13, in <module>
11:43:10 PRODUCER.1           |     from gryphon.data_service.auditor_task import AuditorTask
11:43:10 PRODUCER.1           |   File "/usr/local/lib/python2.7/site-packages/gryphon/data_service/auditor_task.py", line 2, in <module>
11:43:10 PRODUCER.1           |     from gryphon.data_service.auditors.bitstamp_orderbook_auditor import BitstampOrderbookAuditor
11:43:10 PRODUCER.1           |   File "/usr/local/lib/python2.7/site-packages/gryphon/data_service/auditors/bitstamp_orderbook_auditor.py", line 3, in <module>
11:43:10 PRODUCER.1           |     from gryphon.data_service.auditors.orderbook_auditor import OrderbookAuditor
11:43:10 PRODUCER.1           |   File "/usr/local/lib/python2.7/site-packages/gryphon/data_service/auditors/orderbook_auditor.py", line 18, in <module>
11:43:10 PRODUCER.1           |     from gryphon.lib.exchange.base import Exchange
11:43:10 PRODUCER.1           |   File "/usr/local/lib/python2.7/site-packages/pyximport/pyximport.py", line 431, in load_module
11:43:10 PRODUCER.1           |     language_level=self.language_level)
11:43:10 PRODUCER.1           |   File "/usr/local/lib/python2.7/site-packages/pyximport/pyximport.py", line 209, in load_module
11:43:10 PRODUCER.1           |     inplace=build_inplace, language_level=language_level)
11:43:10 PRODUCER.1           |   File "/usr/local/lib/python2.7/site-packages/pyximport/pyximport.py", line 186, in build_module
11:43:10 PRODUCER.1           |     reload_support=pyxargs.reload_support)
11:43:10 PRODUCER.1           |   File "/usr/local/lib/python2.7/site-packages/pyximport/pyxbuild.py", line 104, in pyx_to_dll
11:43:10 PRODUCER.1           |     dist.run_commands()
11:43:10 PRODUCER.1           |   File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 953, in run_commands
11:43:10 PRODUCER.1           |     self.run_command(cmd)
11:43:10 PRODUCER.1           |   File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/dist.py", line 972, in run_command
11:43:10 PRODUCER.1           |     cmd_obj.run()
11:43:10 PRODUCER.1           |   File "/usr/local/lib/python2.7/site-packages/Cython/Distutils/build_ext.py", line 163, in run
11:43:10 PRODUCER.1           |     _build_ext.build_ext.run(self)
11:43:10 PRODUCER.1           |   File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 340, in run
11:43:10 PRODUCER.1           |     self.build_extensions()
11:43:10 PRODUCER.1           |   File "/usr/local/lib/python2.7/site-packages/Cython/Distutils/build_ext.py", line 171, in build_extensions
11:43:10 PRODUCER.1           |     self.build_extension(ext)
11:43:10 PRODUCER.1           |   File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/build_ext.py", line 499, in build_extension
11:43:10 PRODUCER.1           |     depends=ext.depends)
11:43:10 PRODUCER.1           |   File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/ccompiler.py", line 574, in compile
11:43:10 PRODUCER.1           |     self._compile(obj, src, ext, cc_args, extra_postargs, pp_opts)
11:43:10 PRODUCER.1           |   File "/usr/local/Cellar/python@2/2.7.16/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/unixccompiler.py", line 124, in _compile
11:43:10 PRODUCER.1           |     raise CompileError, msg
11:43:10 PRODUCER.1           | ImportError: Building module gryphon.lib.exchange.base failed: ["CompileError: command 'clang' failed with exit status 1\n"]
11:43:11 PRODUCER.1           | exited with code 1
11:43:11 system               | sending SIGTERM to all processes
11:43:11 VOLUMES_CONSUMER.1   | terminated by SIGTERM
11:43:11 TRADES_CONSUMER.1    | terminated by SIGTERM
11:43:11 ORDERBOOK_CONSUMER.1 | terminated by SIGTERM

Thanks in advance for the help ;)

[garethdmm]

Hmm interesting, this might be a pretty deep one or might be pretty simple.

This line is importing the old exchange wrapper base class which is deprecated:

gryphon/gryphon/data_service/auditors/orderbook_auditor.py

Line 18 in 8cd2f43

from gryphon.lib.exchange.base import Exchange

That's strange because it's not even used in that file. You could try commenting that line out or deleting it completely. My guess is you'll at least see a new (more important) error.

In general, to get GDS orderbook data into dashboards you only need the orderbook poll task (to get the orderbooks) and the orderbook consumer (to put them in your gds database). So you could comment out the other producer tasks if they start to misbehave:

gryphon/gryphon/data_service/runt.py

Lines 68 to 71 in 8cd2f43

	TradesPollTask(exchanges=non_orderbook_exchanges).start_task()
	AuditorTask(exchanges=non_orderbook_exchanges).start_task()
	UtilitiesPollTask().start_task()
	VolumePollTask(exchanges=non_orderbook_exchanges).start_task()

I'd love to make GDS more configurable from the command line. I'd like users to be able decide which pollers they are going to run on which exchanges, and likely have the default that you only run the orderbook producer and adding extra producer tasks or consumers are add-ons.

[garethdmm]

This should be fixed in #54. If you have a moment could you pull that branch and verify that you no longer see that compile error?