⚠ Upgrade shub to the latest version via:
pip install shub --upgrade
You can define the requirements that you want to deploy to Scrapinghub via your local project's scrapinghub.yml. If you don't have this file in your project folder, run
shub deploy to generate it.
After that, create a requirements.txt file (if you haven't yet) and add your dependencies to it, one dependency per line:
Note: you should always set the specific version for each of your requirements, as shown in the example above. Check this warning for details.
After creating the file, add the
requirements_file setting to scrapinghub.yml pointing it to your project's requirements.txt path:
Now, when you run
shub deploy again, it will deploy your project's dependencies to Scrapinghub too.
Things to keep in mind
Don't Set Requirements in Editable Mode
please consider changing it to
Specify Each Requirement Version
The build process aggressively cache requirements, so pointing to a non-specific version of your requirement is not a good idea as you can't be sure which version of your code is going to be build.
So this is BAD:
This is GOOD:
And this is faster and BETTER:
You can learn more about
requirements.txt file format here.
This video demonstrates how to deploy dependencies to Scrapy Cloud: