How to get celery working with scrapy server on heroku and django-dynamic-scraper? -

I'm in the process of building my first project involving scary, everything on my development server (windows) Works from, but there are some problems with it, I am using django-dynamic-scraper which operates the integration work for me.

I run the following commands in different command prompts in Windows:

 : scrap server: Python manage.py celeryd -l info: python manage.py celerybeat  

On the Heroes, I run the following:

 : Horoko Bash> Heroco run scrap server (app not found problem): Harukoc run pane manage.py Celeryd -l info -B -settings = myapp.production  

There is no error or problem with the actual DJGenzo app. I can use the admin website. Scrappy server runs: Scrappy Web Console available at:

 : http: //read0.0.0: 6800 / [console]: [Launcher] Scrapered start: max_proc = 16, runner = 'scrapyd.runner' : Starting from site 6800: Starting factory & lt; Twisted.web.server.Site example 0x7f1511f62ab8 & gt;  

And celery beats and workers are working:

 : info / beat] green: starting ... ... info / beat ] Entries are being written ...: INFO / MainProcess] Connected to the DJ: // Guest: ** @ Localhost: 5672 //: Warning / Main Process] Salary @ 081b4100-eb7f-441c-976d-ecf97d2d7e5a Ready. : Info / beet] writing entries ...: info / bit] writing entries ...  

First digit: periodic work to run the spider Trigger I get the following error in celery log.

  file "/app/.heroku/python/lib/python2.7/site-packages/dynamic_scraper/utils/ta sk_utils.py", line 31, _pending_jobs resp = urllib2.urlopen ( 'Http: // localhost: 6800 / listjobs.json? Project = default') ... ... in the file "/app/.heroku/python/lib/python2.7 / urllib2.py", line 1184, do_open Increases the URLRRR URLRRR in: & lt; Urlopen error [error 111] connection refused & gt;  

It seems that for some reason, Heroko Seleri is not allowed to access the Scrapper server.

Here are some of my settings:

Scrap.cfg

  [settings] default = myapp.scraper.scrape.settings [deploy] #url = Http: // localhost: 6800 / project = myapp  

celery config

  [config] app: default: 0x7fd4983f6310 (djcelery.loaders.DjangoL transport : Django: // guest: ** @ localhost: 5672 // Result: Database concurrency: 4 (prefork) [queue] celery exchange = celery (direct) key = celery  

already Thank you and tell me that you need more information.

The answer is: Your web app, celery, and scrap servers can not run on the server and allow them to talk to each, however, there are two ways to complete this setup with Heroki.

Option 1: < / P>

1. "myapp-scrapy.herokuapp.com" for a host on your skaker server.
2. Then your django-scrapy app is called "myapp.herokuapp.com" Deploy to another host.
3. Change all events in django-dynamic-scraper open task_utls.py and from localhost: 6800 to myapp-scrapy.herokuapp.com . Option 2:
   1. Simply use the Heroku's scheduler to call your scraper on the command line manually bypassing all dynamic scheduling attributes. , But for some use cases which are just fine.

   I hope this helps someone save some pain.