Web Scraping and Automation

python

Getting anime info scraped from various websites using selenium-python and send the scraped data directly to any device in the form of pushbullet message using pushbullet API

Contents:


Steps to Reproduce


These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

  1. Get the code by forking and cloning the repository


  1. Download the code t your local machine and open the folder named Script.



  1. Right click on script1.py file and select –> open with IDLE as shown below ( You will get this option if python is already installed in your pc )



  1. Now select ‘Run Module’ from ‘Run’ option as shown in next step ( Make sure that you’re connected to the internet before running the script )

  2. After clicking that one few windows might open like browser, and terminal window as shown below: ( After few seconds the browser will automatically close and you can see your results )



You can use pyCharm to run the script:




Prerequisites

Packages/modules required to run the program.

- selenium package

use pip to install the selenium package. Python 3.6 has pip available in the standard library. Using pip, you can install selenium by opening your powershell and typing the following:

pip install selenium


Setting Up Pushbullet Account

If you want to get the scraped data sent to smartphone or computer in the form of pushbullet message then follow the below steps else skip:

  1. Make a pushbullet account



  1. Make a channel on your pushbullet account



  1. generate your pushbullet ACCESS TOKEN as follows



  1. Copy the ACCESS TOKEN you generated and replace it with “YOUR_ACCESS_TOKEN” in the code in line
api_key = 'YOUR_ACCESS_TOKEN'
  1. Uncomment the lines marked in the following image



  1. Run the module again as shown in step 5 of Steps to Reproduce section


Enhancements

Add, delete or edit code as per requirement to add certain enhancements/features like Custom Search Feature.


You can also add feature like:

  • Season Info
  • When is the next episode of any particular anime is coming?
  • Next episode synopsis
  • Airing or Completed/Finished
  • What are the Genres of any anime?
  • And many more…..


You can also host your python code online on wayScript and schedule it so that it’ll run automatically

wayscript


Custom Search Feature


Added a custom search option by modifying the script so that Ii can look for a particular anime. You can also contribute to the code and add more such features to it.



I’ve provided the code for above in Script2 folder, with custom search code and pushbullet code, for more clarity.


References

  1. pushbullet Docs

  2. selenium Docs

  3. How to use selenium


Authors

  • Piyush Kumar ( styles )



subscribe via RSS