Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

After upgrade to 1.0.6 (from deploy script), Locate fails #26

Open
dwmcqueen opened this issue Feb 14, 2018 · 5 comments
Open

After upgrade to 1.0.6 (from deploy script), Locate fails #26

dwmcqueen opened this issue Feb 14, 2018 · 5 comments

Comments

@dwmcqueen
Copy link
Contributor

When trying to run Locate, I get the following error:

`NFO 2018-02-14 15:34:15,262 | Start task "Locate", id=24
INFO 2018-02-14 15:34:15,270 | Deleted 0 UrlUsage objects
INFO 2018-02-14 15:34:15,274 | Deleted 0 TextUnitTag(tag=url)
INFO 2018-02-14 15:34:15,277 | Deleted 0 DateDurationUsage objects
INFO 2018-02-14 15:34:15,280 | Deleted 0 TextUnitTag(tag=duration)
INFO 2018-02-14 15:34:15,282 | Deleted 0 CourtUsage objects
INFO 2018-02-14 15:34:15,285 | Deleted 0 TextUnitTag(tag=court)
INFO 2018-02-14 15:34:15,288 | Deleted 0 CopyrightUsage objects
INFO 2018-02-14 15:34:15,290 | Deleted 0 TextUnitTag(tag=copyright)
INFO 2018-02-14 15:34:15,293 | Deleted 0 DefinitionUsage objects
INFO 2018-02-14 15:34:15,296 | Deleted 0 TextUnitTag(tag=definition)
INFO 2018-02-14 15:34:15,298 | Deleted 0 RatioUsage objects
INFO 2018-02-14 15:34:15,301 | Deleted 0 TextUnitTag(tag=ratio)
INFO 2018-02-14 15:34:15,304 | Deleted 0 CurrencyUsage objects
INFO 2018-02-14 15:34:15,306 | Deleted 0 TextUnitTag(tag=currency)
INFO 2018-02-14 15:34:15,309 | Deleted 0 PartyUsage objects
INFO 2018-02-14 15:34:15,311 | Deleted 0 TextUnitTag(tag=party)
INFO 2018-02-14 15:34:15,314 | Deleted 0 DateUsage objects
INFO 2018-02-14 15:34:15,317 | Deleted 0 TextUnitTag(tag=date)
INFO 2018-02-14 15:34:15,320 | Deleted 0 TrademarkUsage objects
INFO 2018-02-14 15:34:15,324 | Deleted 0 TextUnitTag(tag=trademark)
INFO 2018-02-14 15:34:15,327 | Deleted 0 DistanceUsage objects
INFO 2018-02-14 15:34:15,330 | Deleted 0 TextUnitTag(tag=distance)
INFO 2018-02-14 15:34:15,332 | Deleted 0 PercentUsage objects
INFO 2018-02-14 15:34:15,335 | Deleted 0 TextUnitTag(tag=percent)
INFO 2018-02-14 15:34:15,338 | Deleted 0 AmountUsage objects
INFO 2018-02-14 15:34:15,340 | Deleted 0 TextUnitTag(tag=amount)
INFO 2018-02-14 15:34:15,343 | Deleted 0 GeoEntityUsage objects
INFO 2018-02-14 15:34:15,345 | Deleted 0 GeoAliasUsage objects
INFO 2018-02-14 15:34:15,348 | Deleted 0 TextUnitTag(tag=geoentity)
INFO 2018-02-14 15:34:15,351 | Deleted 0 CitationUsage objects
INFO 2018-02-14 15:34:15,354 | Deleted 0 TextUnitTag(tag=citation)
INFO 2018-02-14 15:34:15,356 | Deleted 0 RegulationUsage objects
INFO 2018-02-14 15:34:15,359 | Deleted 0 TextUnitTag(tag=regulation)
INFO 2018-02-14 15:34:15,362 | Deleted 0 TermUsage objects
INFO 2018-02-14 15:34:15,365 | Deleted 0 TextUnitTag(tag=term)
INFO 2018-02-14 15:34:23,614 | Run location of [url; duration; court; definition; ratio; distance; party; date; trademark; copyright; percent; amount; geoentity; currency; citation; regulation; term].
INFO 2018-02-14 15:34:23,617 | Locate in [paragraphs and sentences].
INFO 2018-02-14 15:34:23,629 | Found 39238 Text Units. Added 39238 subtasks.
ERROR 2018-02-14 15:35:09,604 | ERROR
ERROR 2018-02-14 15:35:09,638 | OperationalError: [Errno 111] Connection refused
ERROR 2018-02-14 15:35:09,640 | Traceback (most recent call last):
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 494, in _ensured
return fun(*args, **kwargs)
File "/opt/ve/lib/python3.5/site-packages/kombu/messaging.py", line 203, in _publish
mandatory=mandatory, immediate=immediate,
File "/opt/ve/lib/python3.5/site-packages/amqp/channel.py", line 1734, in _basic_publish
(0, exchange, routing_key, mandatory, immediate), msg
File "/opt/ve/lib/python3.5/site-packages/amqp/abstract_channel.py", line 50, in send_method
conn.frame_writer(1, self.channel_id, sig, args, content)
File "/opt/ve/lib/python3.5/site-packages/amqp/method_framing.py", line 166, in write_frame
write(view[:offset])
File "/opt/ve/lib/python3.5/site-packages/amqp/transport.py", line 258, in write
self._write(s)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 414, in _reraise_as_library_errors
yield
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 515, in _ensured
reraise_as_library_errors=False,
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 405, in ensure_connection
callback)
File "/opt/ve/lib/python3.5/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
return fun(*args, **kwargs)
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 261, in connect
return self.connection
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 802, in connection
self._connection = self._establish_connection()
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 757, in _establish_connection
conn = self.transport.establish_connection()
File "/opt/ve/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/opt/ve/lib/python3.5/site-packages/amqp/connection.py", line 282, in connect
self.transport.connect()
File "/opt/ve/lib/python3.5/site-packages/amqp/transport.py", line 109, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/opt/ve/lib/python3.5/site-packages/amqp/transport.py", line 150, in _connect
self.sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/lexpredict-contraxsuite/contraxsuite_services/apps/task/tasks.py", line 210, in run
self.process(**kwargs)
File "/opt/lexpredict-contraxsuite/contraxsuite_services/apps/task/tasks.py", line 822, in process
task_id='%d_%s' % (self.task.id, fast_uuid()))
File "/opt/ve/lib/python3.5/site-packages/celery/app/task.py", line 536, in apply_async
**options
File "/opt/ve/lib/python3.5/site-packages/celery/app/base.py", line 737, in send_task
amqp.send_task_message(P, name, message, **options)
File "/opt/ve/lib/python3.5/site-packages/celery/app/amqp.py", line 554, in send_task_message
**properties
File "/opt/ve/lib/python3.5/site-packages/kombu/messaging.py", line 181, in publish
exchange_name, declare,
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 527, in _ensured
errback and errback(exc, 0)
File "/usr/lib/python3.5/contextlib.py", line 77, in exit
self.gen.throw(type, value, traceback)
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 419, in _reraise_as_library_errors
sys.exc_info()[2])
File "/opt/ve/lib/python3.5/site-packages/vine/five.py", line 178, in reraise
raise value.with_traceback(tb)
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 414, in _reraise_as_library_errors
yield
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 515, in _ensured
reraise_as_library_errors=False,
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 405, in ensure_connection
callback)
File "/opt/ve/lib/python3.5/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
return fun(*args, **kwargs)
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 261, in connect
return self.connection
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 802, in connection
self._connection = self._establish_connection()
File "/opt/ve/lib/python3.5/site-packages/kombu/connection.py", line 757, in _establish_connection
conn = self.transport.establish_connection()
File "/opt/ve/lib/python3.5/site-packages/kombu/transport/pyamqp.py", line 130, in establish_connection
conn.connect()
File "/opt/ve/lib/python3.5/site-packages/amqp/connection.py", line 282, in connect
self.transport.connect()
File "/opt/ve/lib/python3.5/site-packages/amqp/transport.py", line 109, in connect
self._connect(self.host, self.port, self.connect_timeout)
File "/opt/ve/lib/python3.5/site-packages/amqp/transport.py", line 150, in _connect
self.sock.connect(sa)
kombu.exceptions.OperationalError: [Errno 111] Connection refused

INFO 2018-02-14 15:35:09,646 | End of main task "Locate", id=24
INFO 2018-02-14 15:35:09,654 | There are 39001 uncompleted subtasks remaining. Please wait.`
@dwmcqueen
Copy link
Contributor Author

I upgraded to 1.06 by git pull ing the latest deploy (to add in new fabricrc and python requirements) and then ran "update-local.sh" as the local (non-root) user. The script appeared to run correctly and the site loaded (with 1.0.6 indicated as the new version).

@reddalexx
Copy link
Contributor

Such errors usually mean that celery doesn't work.
possible solution:

  1. check that celery works: celery status -A apps
  2. restart it if needed: celery multi restart 2 -A apps -f celery.log -B
  3. check that rabbitmg works: sudo rabbitmqctl status
    then restart if needed: sudo rabbitmqctl restart

but as I think restarting celery should fix the issue
if it doesn't help I need logs: celery.log, /var/log/uwsgi/contrax.log

@dwmcqueen
Copy link
Contributor Author

It looks like Celery is running as well as rabbitmq. Attached are the files you mentioned,
celery.log
contrax.log

@reddalexx
Copy link
Contributor

Perhaps it didn't work at 15:35:09
but as I see from celery.log celery works good since that time
[2018-02-14 15:35:29,106: INFO/MainProcess] Connected to amqp://contrax1:@127.0.0.1:5672/contrax1_vhost
[2018-02-14 15:35:29,108: INFO/MainProcess] Connected to amqp://contrax1:@127.0.0.1:5672/contrax1_vhost

may be there was some issue with rabbitmq in that time, at least celery.log says that
[2018-02-14 15:35:29,093: WARNING/MainProcess] consumer: Connection to broker lost. Trying to re-establish the connection...

@dwmcqueen
Copy link
Contributor Author

Something did happen and now tasks are hung. I think I am going to wipe the server and move to the docker deployment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dwmcqueen @reddalexx