How To Scrape LinkedIn Public Company Data – Beginners Guide

How To Scrape LinkedIn Public Company Data

Nowadays everybody is familiar with how big the LinkedIn community is. LinkedIn is one of the largest professional social networking sites in the world which holds a wealth of information about industry insights, data on professionals, and job data.

Now, the only way to get the entire data out of LinkedIn is through Web Scraping.

Why Scrape LinkedIn public data?

There are multiple reasons why one wants to scrape the data out of LinkedIn. The scrape data can be useful when you are associated with the project or for hiring multiple people based on their profile while looking at their data and selecting among them who all are applicable and fits for the company best.

This scraping task will be less time-consuming and will automate the process of searching for millions of data in a single file which will make the task easy.

Another benefit of scraping is when one wants to automate their job search. As every online site has thousands of job openings for different kinds of jobs, so it must be hectic for people who are looking for a job in their field only. So scraping can help them automate their job search by applying filters and extracting all the information at only one page.

In this tutorial, we will be scraping the data from LinkedIn using Python.


In this tutorial, we will use basic Python programming as well as some python packages- LXML and requests.

But first, you need to install the following things:

  1. Python accessible here (
  2. Python requests accessible here(
  3. Python LXML( Study how to install it here:

Once you are done with installing here, we will write the python code to extract the LinkedIn public data from company pages.

This below code will only run on python 2 and not above them because the sys function is not supported in it.

import json

import re

from importlib import reload

import lxml.html

import requests

import sys



HEADERS = {'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',

          'accept-encoding': 'gzip, deflate, sdch',

          'accept-language': 'en-US,en;q=0.8',

          'upgrade-insecure-requests': '1',

          'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.85 Safari/537.36'}

file = open('company_data.json', 'w')




def increment():

   global COUNT


def fetch_request(url):


       fetch_url = requests.get(url, headers=HEADERS)



           fetch_url = requests.get(url, headers=HEADERS)



               fetch_url = requests.get(url, headers=HEADERS)


               fetch_url = ''

   return fetch_url

def parse_company_urls(company_url):

   if company_url:

       if '/company/' in company_url:



           parent_url = company_url


           if fetch_company_url:

               sel = lxml.html.fromstring(fetch_company_url.content)

               COMPANIES_XPATH = '//div[@class="section last"]/div/ul/li/a/@href'

               companies_urls = sel.xpath(COMPANIES_XPATH)

               if companies_urls:

                   if '/company/' in companies_urls[0]:

                       print('Parsing From Category ', parent_url)


                   for company_url in companies_urls:




def parse_company_data(company_data_url):

   if company_data_url:

       fetch_company_data = fetch_request(company_data_url)

       if fetch_company_data.status_code == 200:


               source = fetch_company_data.content.decode('utf-8')

               sel = lxml.html.fromstring(source)

               # CODE_XPATH = '//code[@id="stream-promo-top-bar-embed-id-content"]'

               # code_text = sel.xpath(CODE_XPATH).re(r'<!--(.*)-->')

               code_text = sel.get_element_by_id(


               if len(code_text) > 0:

                   code_text = str(code_text[0])

                   code_text = re.findall(r'<!--(.*)-->', str(code_text))

                   code_text = code_text[0].strip() if code_text else '{}'

                   json_data = json.loads(code_text)

                   if json_data.get('squareLogo', ''):

                       company_pic = '' + \

                                     json_data.get('squareLogo', '')

                   elif json_data.get('legacyLogo', ''):

                       company_pic = '' + \

                                     json_data.get('legacyLogo', '')


                       company_pic = ''

                   company_name = json_data.get('companyName', '')

                   followers = str(json_data.get('followerCount', ''))

                   # CODE_XPATH = '//code[@id="stream-about-section-embed-id-content"]'

                   # code_text = sel.xpath(CODE_XPATH).re(r'<!--(.*)-->')

                   code_text = sel.get_element_by_id(


               if len(code_text) > 0:

                   code_text = str(code_text[0]).encode('utf-8')

                   code_text = re.findall(r'<!--(.*)-->', str(code_text))

                   code_text = code_text[0].strip() if code_text else '{}'

                   json_data = json.loads(code_text)

                   company_industry = json_data.get('industry', '')

                   item = {'company_name': str(company_name.encode('utf-8')),

                           'followers': str(followers),

                           'company_industry': str(company_industry.encode('utf-8')),

                           'logo_url': str(company_pic),

                           'url': str(company_data_url.encode('utf-8')), }



                   file = open('company_data.json', 'a')






fetch_company_dir = fetch_request('')

if fetch_company_dir:

   print('Starting Company Url Scraping')


   sel = lxml.html.fromstring(fetch_company_dir.content)

   SUB_PAGES_XPATH = '//div[@class="bucket-list-container"]/ol/li/a/@href'

   sub_pages = sel.xpath(SUB_PAGES_XPATH)

   print('Company Category URL list')



   if sub_pages:

       for sub_page in sub_pages:




How To Scrape Amazon Data Using Python Scrapy

How To Scrape Amazon Data Using Python Scrapy

Will it not be good if all the information related to some product will be placed in only one table? I guess it will be really awesome and accessible if we can get the entire information at one place.

Since, Amazon is a huge website containing millions of data so scraping the data is quite challenging. Amazon is a tough website to scrape for beginners and people often get blocked by Amazon’s anti-scraping technology.

In this blog, we will be aiming to provide the information about the scrapy and how to scrape the Amazon website using it.

What is Scrapy?

Scrapy is a free and open-source web-crawling Python’s framework. It was originally designed for web scraping, extracting the data using API’s and or general-purpose web crawler.

This framework is used in data mining, information processing or historical archival. The applications of this framework is used widely in different industries and has been proven very useful. It not only scrapes the data from the website, but it is able to scrape the data from the web services also. For example, Amazon API, Facebook API, and many more.

How to install Scrapy?

Firstly, there are some third-party softwares which needs to be installed in order to install the Scrapy module.

  • Python: As Scrapy has the base of the Python language, one has to install it first.
  • pip: pip is a python package manager tool which maintains a package repository and installs python libraries, and its dependencies automatically. It is better to install pip according to system OS, and then try to follow the standard way of installing Scrapy.

There are different ways in which we can download Scrapy globally as well as locally but the most standard way of downloading it is by using pip.

Run the below command to install Scrapy using pip:

Pip install scrapy

How to get started with Scrapy?

Since we know that Scrapy is an application framework and it provides multiple commands to create an application and use them. But before everything, we have to set up a new Scrapy project. Enter a directory where you’d like to store your code and run:

Scrapy startproject new_project

This will create a directory:

Scrapy is an application framework which follows object oriented programming style for the definition of items and spiders for overall applications.

The project structure contains different the following files:

  1. Scrapy.cfg : This file is a root directory of the project, which includes project name with the project settings.
  2. Test_project : It is an application directory with many different files which actually make running and scraping responsible from the web URLs.
  3. :Items are containers that will be loaded with the scraped data and they work like simple python dictionaries.Items provide additional prevention against typos and populating undeclared fields.
  4. :After the scraping of an item has been done by the spider, it is sent to the item pipeline which processes it through several components. Each class has to implement a method called process_item for the processing of scraped items.
  5. : It allows the customization of the behaviour of all scrapy components, including the core, extensions, pipelines, and spiders themselves.
  6. Spiders : It is a directory which contains all spiders/crawlers as python classes.

Scrape Amazon Data: How to Scrape an Amazon Web Page

For a better understanding of how the scrapy works, we will be scraping the product name, price, category, and it’s availability from the website.

Let’s name this project amazon_pro. You can use the project name according to your choice.

Start by writing the below code:

Scrapy startproject test_project

The directory will be created in the local folder by the name mentioned above.

Now we need three things which will help in the scraping process.

  1. Update field which we want to scrape. For example names, price, availability, and so on.
  2. We have to create a new spider with all the necessary elements in it like allowed domains, start_urls and parse method.
  3. For data processing, we have to update file.

Now after creating the spider, follow thee further steps given in the terminal:

  1. Scrapy genspider amazon

Now, we need to define the name, URLs, and possible domains to scrape the data.

How to Scrape an Amazon Web Page 2

An item object is defined in the parse method and is filled with required information using the utility of XPath response object. It is a search function that is used to find elements in the HTML tree structure. Lastly let’s yield the items object, so that scrapy can do further processing on it.

Next, after scraping data, scrapy calls Item pipelines to process them. These are called Pipeline classes and we can use these classes to store data in a file or database or in any other way. It is a default class like Items that scrapy generates for users.

How to Scrape an Amazon Web Page 3

The process_item is implemented by the pipeline classes method and items are being yielded by a Spider each and every time. It takes the item and spider class as arguments and returns a dict object. So for this example, we are just returning item dict as it is.

Now, we have to enable the ITEM_PIPELINES from file.

How to Scrape an Amazon Web Page 4

Now, after completing the entire code, we need to scrape the item by sending requests and accepting response objects.

We will call a spider by its unique name and scrapy will easily search from it.

Scrapy crawl amazon

Now, after the items have been scraped, we can save it to different formats using their extensions. For example, .json, .csv, and many more.

Scrapy crawl amazon -o data.csv

The above command will save the scraped data in the csv format in data.csv file.

Here is the output of the above code:

{"product_category": "Electronics,Computers & Accessories,Data Storage,External Hard Drives", "product_sale_price": "$949.95", "product_name": "G-Technology G-SPEED eS PRO High-Performance Fail-Safe RAID Solution for HD/2K Production 8TB (0G01873)", "product_availability": "Only 1 left in stock."},
{"product_category": "Electronics,Computers & Accessories,Data Storage,USB Flash Drives", "product_sale_price": "", "product_name": "G-Technology G-RAID with Removable Drives High-Performance Storage System 4TB (Gen7) (0G03240)", "product_availability": "Available from these sellers."},
{"product_category": "Electronics,Computers & Accessories,Data Storage,USB Flash Drives", "product_sale_price": "$549.95", "product_name": "G-Technology G-RAID USB Removable Dual Drive Storage System 8TB (0G04069)", "product_availability": "Only 1 left in stock."},
{"product_category": "Electronics,Computers & Accessories,Data Storage,External Hard Drives", "product_sale_price": "$89.95", "product_name": "G-Technology G-DRIVE ev USB 3.0 Hard Drive 500GB (0G02727)", "product_availability": "Only 1 left in stock."}

We have successfully scraped the data from using scrapy.

How to Code a Scraping Bot with Selenium and Python

How to Code a Scraping Bot with Selenium and Python

Selenium is a powerful tool for controlling web browsers through programs and performing browser automation. Selenium is also used in python for scraping the data. It is also useful for interacting with the page before collecting the data, this is the case that we will discuss in this article.

In this article, we will be scraping the to extract the historical data of dollar exchange rates against one or more currencies.

There are other tools in python by which we can extract the financial information. However, here we want to explore how selenium helps with data extraction.

The Website we are going to Scrape:

Understanding of the website is the initial step before moving on to further things.

Website consists of historical data for the exchange rate of dollars against euros.

In this page, we will find a table in which we can set the date range which we want.

That is the thing which we will be using.

We only want the currencies exchange rate against the dollar. If that’s not the case then replace the “usd” in the URL.

The Scraper’s Code:

The initial step is starting with the imports from the selenium, the Sleep function to pause the code for some time and the pandas to manipulate the data whenever necessary.

How to Code a Scraping Bot with Selenium and Python

Now, we will write the scraping function. The function will consists of:

  • A list of currency codes.
  • A start date.
  • An End date.
  • A boolean function to export the data into .csv file. We will be using False as a default.

We want to make a scraper that scrapes the data about the multiple currencies. We also have to initialise the empty list to store the scraped data.

How to Code a Scraping Bot with Selenium and Python 1

As we can see that the function has the list of currencies and our plan is to iterate over this list and get the data.

For each currency we will create a URL, instantiate the driver object, and we will get the page by using it.

Then the window function will be maximized but it will only be visible when we will keep the option.headless as False.

Otherwise, all the work will be done by the selenium without even showing you.

How to Code a Scraping Bot with Selenium and Python 2

Now, we want to get the data for any time period.

Selenium provides some awesome functionalities for getting connected to the website.

We will click on the date and fill the start date and end dates with the dates we want and then we will hit apply.

We will use WebDriverWait, ExpectedConditions, and By to make sure that the driver will wait for the elements we want to interact with.

The waiting time is 20 seconds, but it is to you whichever the way you want to set it.

We have to select the date button and it’s XPath.

The same process will be followed by the start_bar, end_bar, and apply_button.

The start_date field will take in the date from which we want the data.

End_bar will select the date till which we want the data.

When we will be done with this, then the apply_button will come into work.

How to Code a Scraping Bot with Selenium and Python 3

Now, we will use the pandas.read_html file to get all the content of the page. The source code of the page will be revealed and then finally we will quit the driver.

How to Code a Scraping Bot with Selenium and Python 4

How to handle Exceptions In Selenium:

The collecting data process is done. But selenium is sometimes a little unstable and fail to perform the function we are performing here.

To prevent this we have to put the code in the try and except block so that every time it faces any problem the except block will be executed.

So, the code will be like:

for currency in currencies:

        while True:


                # Opening the connection and grabbing the page

                my_url = f'{currency.lower()}-historical-data'

                option = Options()

                option.headless = False

                driver = webdriver.Chrome(options=option)




                # Clicking on the date button

                date_button = WebDriverWait(driver, 20).until(






                # Sending the start date

                start_bar = WebDriverWait(driver, 20).until(






                # Sending the end date

                end_bar = WebDriverWait(driver, 20).until(







                # Clicking on the apply button

                apply_button = WebDriverWait(driver,20).until(







                # Getting the tables on the page and quiting

                dataframes = pd.read_html(driver.page_source)


                print(f'{currency} scraped.')





                print(f'Failed to scrape {currency}. Trying again in 30 seconds.')



For each DataFrame in this dataframes list, we will check if the name matches, Now we will append this dataframe to the list we assigned in the beginning.

Then we will need to export a csv file. This will be the last step and then we will be over with the extraction.

How to Code a Scraping Bot with Selenium and Python 5

Wrapping up:

This is all about extracting the data from the website.So far this code gets the historical data of the exchange rate of a list of currencies against the dollar and returns a list of DataFrames and several .csv files.

How to Scrape Wikipedia Articles with Python

How to Scrape Wikipedia Articles with Python

We are going to make a scraper which will scrape the wikipedia page.

The scraper will get directed to the wikipedia page and then it will go to the random link.

I guess it will be fun looking at the pages the scraper will go.

Setting up the scraper:

Here, I will be using Google Colaboratory, but you can use Pycharm or any other platform you want to do your python coding.

I will be making a colaboratory named Wikipedia. If you will use any python platform then you need to create a .py file followed by the any name for your file.

To make the HTTP requests, we will be installing the requests module available in python.

Pip install requests

 We will be using a wiki page for the starting point.

Import requests

Response = requests.get(url = “”)


 When we run the above command, it will show 200 as a status code.

How to Scrape Wikipedia Articles with Python 1

Okay!!! Now we are ready to step on the next thing!!

Extracting the data from the page:

We will be using beautifulsoup to make our task easier. Initial step is to install the beautiful soup.

Pip install beautifulsoup4

Beautiful soup allows you to find an element by the ID tag.

Title = soup.find( id=”firstHeading”)

 Bringing everything together, our code will look like:

How to Scrape Wikipedia Articles with Python 3

As we can see, when the program is run, the output is the title of the wiki article i.e Web Scraping.

 Scraping other links:

Other than scraping the title of the article, now we will be focusing on the rest of the things we want.

We will be grabbing <a> tag to another wikipedia article and scrape that page.

To do this, we will scrape all the <a> tags within the article and then I will shuffle it.

Do not forget to import the random module.

How to Scrape Wikipedia Articles with Python 3

You can see, the link is directed to some other wikipedia article page named as IP address.

Creating an endless scraper:

Now, we have to make the scraper scrape the new links.

For doing this, we have to move everything into scrapeWikiArticle function.

How to Scrape Wikipedia Articles with Python 4

The function scrapeWikiArticle will extract the links and and title. Then again it will call this function and will create an endless cycle of scrapers that bounce around the wikipedia.

After running the program, we got:

How to Scrape Wikipedia Articles with Python 5

Wonderful! In only few steps, we got the “web scraping” to “Wikipedia articles with NLK identifiers”.


We hope that this article is useful to you and you learned how to extract random wikipedia pages. It revolves around wikipedia by following random links.

Python Data Persistence – A Quick Guide

Almost every other computer application, whether it is a web based application, a standalone data logger, a mobile app or a desktop application with or without GUI, stores and retrieves data from some persistent storage device such as hard disk or a flash drive. Such storage device may either be connected to computer or it may be available on a network. Without this ability to recurrently access, update and retrieve stored data, most computer applications would have been reduced to programmable calculators!

Python Data Persistence – A Quick Guide

Data storage format depends on the logical structure of data and on the processing logic. Data may be stored in flat computer files, in tables of relational databases or different store formats of NOSQL databases. You will know more about these terms in subsequent chapters of this Page.

Back-end process of a computer application stores, modifies and retrieves data in response to front-end user’s requirements. Almost every programming language offers tools to interact with files/databases. This book aims to familiarize the reader with Python’s functions and modules that handle persistent data processing.

Python Data Persistence – Getting Started

Python Data Persistence – Program Flow Control

Python Data Persistence – Structured Python

Python Data Persistence Object Oriented Programming OOP

Python Data Persistence – File IO

Python Data Persistence – Object Serialization

Python Data Persistence – RDBMS Concepts

Python Data Persistence – Python DB-API

Python Data Persistence – Python – SQLAlchemy

Python Data Persistence – Python and Excel

Python Data Persistence – Python – PyMongo

Python Data Persistence – Python – Cassandra

Read Also: 

Why Python?

Popularity of Python has increased by many fold recently because of the emergence of powerful libraries for data analysis, visualization and machine learning. These libraries use data stored in different formats such as text files and relational databases. Hence to be a proficient data scientist, it is important to have a sound understanding of Python tools for data persistence.

Features of Python

Python is easy!: So why has Python been so popular? First and foremost, Python is very easy to learn and use. “Simple is better than Complex”. This is one of the guiding principles of Python’s design philosophy. It has clean and simple syntax resembling to a natural language. It saves a lot of development time.

Open source: Python is free and open source having a very active and supportive developer community. As a result of plenty of documentation resources, guides tutorials and forums me available in public domain. Any newcomer can access, learn, collaborate and seek advice from community.

Object oriented: Python is completely object oriented, although it supports multiple programming paradigms including functional programming style.

Extensible: Python can be easily integrated with other languages such as C/C++, Java, .NET, etc. Corporate support: Python enjoys great corporate support. Google for example promotes Python in a big way as Python is extensively used in many of its applications. Many other companies use Python as their development platform.

What is Python: Installation Guide | How to Install Python3 & Python2 on Windows, Linux, macOS?

What is Python Installation Guide

Python is the most famous programming language used in various fields. It has two major versions ie., Python 2 and Python 3. To use the Python language on your devices, it is mandatory to install the required version by following all instructions. Python beginners will face some hurdles in installing python in their Operating systems. So, we have come up with the Python Installation tutorial for beginners and experts to use python on their OS. Here, you will learn how to check the current python version, how to install python on Windows, Linux, macOS, and many other.

In this tutorial of what is python installation guide, you’ll learn the basic instructions that you should aware of:

Introduction to Python Installation Guide

Python is a very versatile language. It has thousands of libraries and modules to work with. As a beginner, you have to think that it is very important to start out with a language. It covers a wide range of functions to deal with basic to complex programs.

No matter what operating system you had, we have covered all python installation steps for Windows, macOS, Linux, etc. Installing or updating or setting up python on your device is the initial step to becoming a python programmer. You can discover various methods to install python on different OS. So, checking the python version before downloading or performing the code in python is a must.


If you’re a beginner to Python ensure to read all the tutorials covered by BtechGeeks and get some idea about the language. It is compatible & flexible with all different operating systems. In fact, the Python programming language is easy to set up. In case, you’re using the Linux system you will have python pre-installed on your OS.

How to check the Python version?

Some operating systems come with Python pre-installed Like macOS or Ubuntu. Other operating systems like Windows don’t have it installed out of the box.

In order to check whether Python is installed or not in open the terminal in macOS or Linux. Write the below command:

python --version


Python 3.7.2

If you’re using windows. Open the CMD and write:

python -v

Another way to check the Python version is to write python. It should open the Python shell or raise an error.
How to check the Python version

If your operating system doesn’t have Python installed.

How to download Python?

One of the best features of Python is its compatibility. It can be easily installed in any operating system. You can install it via command line or GUI.

Python installation in Linux-based operating systems is very easy. In this case, we will check the installation for Ubuntu. It’s almost the same in other Linux distributions.

  1. Install dependencies.
    $ sudo apt-get install software-properties-common
    $ sudo add-apt-repository ppa:deadsnakes/ppa
  2. Update the OS package manager
    sudo apt-get update
  3. Install the latest version
    $ sudo apt-get install python3.8

Note that the version of python will depend on your distribution. For example, installing a Python3.6 the package on Fedora 25 to get Python 3.6.

How to Install Python on windows?

  1. Visit the official website.
  2. Navigate to downloads > windows.
  3. Select your version.
  4. Open the .exe file.
  5. Click install now.
  6. Proceed with next.
  7. Finish the installation and click close.

How to download Python How to download Python 1

How to install Python on macOS?

  1. Do the above steps
  2. From the downloads section select MacOS X
  3. Click on the macOS installer.
  4. Open the downloaded file.
  5. Click to continue with recommended settings.
  6. After finish, you will have it installed.

How to install Python in macOS

Python Installation Tutorial for Linux

One of the best open-source OS is Linux. Also, you can see various Linux-based operating systems. The most commonly used are Ubuntu, Fedora, Linux Mint, Debian. Here, we are going to learn how to perform python installation for both versions 3.x and 2.x. Fedora Linux OS used for Installation of python. Almost all the latest Linux OS have already downloaded Python. You can verify it is installed or not by giving the below commands in the terminal.

For Python 2

$ python2 --version

For Python 3

$ python3 --version

If you saw the python versions as output then you are not allowed to install the python. If no, then you have to install python by following the steps provided here to install successfully:

First, you have to open the terminal or Command prompt from your linux based OS and type the following commands:

$ sudo apt-get update

$ sudo apt-get install python3.6


$ sudo apt-get update

$ sudo apt-get install python2.7

If you are using Ubuntu 16.0 or a newer version, then you can easily install Python 3.6 or Python 2.7 by typing the above commands.

What is the difference between Python2.x and 3.x?

If you’re trying to start learning  Python, you’ll find two versions. The 2.x and the 3.x. What is the difference?

Python is like any other language that has versions. The goal of versioning is to track the updates. Each version has its own features. Why are we versioning the updates of any language?

The goal is to be able to track the updates. In the software industry, updating the language can have a side effect. The update can have a deprecated feature or syntax change. An auto-update can lead to project failure. Versioning helps us to specify a version that our project will run with it. This leads to less error and increases the lifetime of the project.

The version number consists of two parts. Like this (2.7 or 3.8). The first number is referring to a Major change. Change that could lead to deprecate some feature or syntax change. The second number is referring to a minor update or fix. Each update has a changelog. With that changelog, developers can track the changes to update their projects. It’s always recommended to work with the latest version of the language.

In Python, there are two versions 2.x and 3.x. The difference is major. Updating the 2.x project to 3.x will lead to a syntax error. What is the difference between 2.x and 3.x? Which one you should learn?

Integer division:

Divide two integers in 2.x  will not have a float value.

# Python 2.x
print 5 / 2

# 2

# Python 3.x
print(5 / 2)

# 2.5

Print function:

The print function in 3.x brackets is mandatory. The print function in Python2.x brackets are optional.

# Python 2.x
print "Hi weclome to python 2.x"

# Hi weclome to python 2.x

# Python 3.x
print("Hi weclome to python 3.x")

# Hi weclome to python 3.x


In 2.x the implicit string type is ASCII. But with 3.x  it’s UNICODE.

print('sample word')
print(b'sample word')

# Python 2.x 
# <type 'str'>
# <type 'str'>

# Python 3.x
# <class 'str'>
# <class 'bytes'>

In 2.x both are the same type. But In 3.x they are different types. This doesn’t mean that Python2.x doesn’t support Unicode. It supports Unicode with different syntax print('this is a text ') .

Range and xrange:

The statement xrange is deprecated in Python3.x. You can check our full article about the range function.

# Python 2.x
for x in xrange(1, 5):

# 1 2 3 4

# Python 3.x
for x in xrange(1, 5):

Traceback (most recent call last):     
  File "<stdin>", line 1, in <module>  
NameError: name 'xrange' is not defined

It should be
for x in range(1, 5):

# 1 2 3 4

Error handling:

Error exception handling in Python3.x should be defined with as.

# Python 2.x
except NameError, err:
    print err, 'ops defined before assign.'

# output:
(NameError("name 'variable_name' is not defined",), 'ops defined before assign.')

# Python 3.x
except NameError, err:
    print err, 'ops defined before assign.'

 File "<ipython-input-1-942e89fbf9ab>", line 3
    except NameError, err:
SyntaxError: invalid syntax

It should be:
except NameError as err:
    print(err, 'ops defined before assign.')

Note: Python2.7 support ended. If you’re new to python It’s recommended to start with Python3.x.

How to set up a ready python environment?

The different versions of python make it not practical to have a single setup for all projects. If you have a python2.x project, it’ll not work on your python3.x environment. You need to isolate each project with its packages. This concept is called a virtual environment. It’s an isolated environment with a specific version to run your project. Using a virtual environment will help you work with different projects easily. All you need it to activate the environment.

How to set up a virtual environment?
Installing a virtual environment ubuntu

How to set up a virtual environment?

Installing a virtual environment ubuntu

  1. Make sure you have it is installed.
  2. Update your package manager
    sudo apt update
  3. Install the pip package manager.
    $ sudo apt install -y python3-pip
  4. Install the essentials tools.
    $ sudo apt install -y build-essential libssl-dev libffi-dev python3-dev
  5. Installing the virtual environment.
    $ sudo apt install -y python3-venv
  6. Create a virtual env.
    $ python3 -m venv <env_name>
  7. Activate the virtual env.
    $ source <env_name>/bin/activate

Now you have an isolated playground. Any installed package will be in this virtual environment. If you are done you can simply write deactivate to deactivate the environment.

Final Thoughts

You can quickly install python on any operating system by following the instructions prevailing in this tutorial. With its versioning system, you can pick the right version for your project. It’s always advised to work with the newest version. You must know the differences between python versions before deciding to choose the python project. Updating the project depends on what version you’re using. It’s suggested to use a virtual environment for each project. It supports you to run multiple projects on the same machine.

How is Python Used in Education? | Python in Education | 5 Reasons Why Python is a Popular Teaching Language

How is python used in education

Present we have plenty of options in programming languages that help to offer in many roles. But the most common question is which is an excellent artificial language for teaching programming basics for all students in schools and colleges. Python in education industries become a present trend. Numerous universities are selecting a python programming language for teaching basic programming courses. Thus, the education industry utilizing this python language in different ways.

Python is a unique tool to utilize for anyone who needs scripts written easily and fast, stated by Forbes. In this Tutorial on How Used Python in Education, we will look at some specific use case situations that address Python as the perfect language for educational purposes.

Python in Education: A Large Collection of Resources

Python is an open-source project that is popular across the world and incredibly simple to learn and understand. It highlights a clear syntax that produces it easy to find out. As a result, there are many resources available for students who preferred to learn the language. Teachers can surely refer their students to additional texts.

Likewise, students have access to materials online that they can utilize for learning. If assignments can’t be solved quickly, students can easily check online for programming assignment guidance. The language also has a large community of programmers more than willing to help you. Whereas, the education industry uses Python and is easier than other artificial languages like Java or C++.

Scripting and Automation

Automation is one of the most significant applications of Python today. As a scripting language, the educational industry can use Python in automating many processes today. Processes like attendance that are repetitive and require fewer judgments can be handled conveniently with Python. That way, teachers can have more time to attend to better things.

Building Computer Graphics

Presently, many offline and online projects use Python to create various Graphical User Interfaces (GUI). Similarly, the education industry can use Python to build beautiful graphics for its software. Even further, developers can use other Python GUIs to produce easier ways to fit in several applications.

Machine Learning and Artificial Intelligence

Today, Python is extensively used in Artificial intelligence and machine learning ahead of all other world programming languages. That is because the algorithms in Python use data information that allows computer systems to perform various functions.

In the Educational industry, teachers and students can use Python in performing some of these functions. For instance, Keras, a library in Python, can facilitate network experimentation. There is also TensorFlow, with which developers can train computer systems to handle tasks that require judgments.

Portable and Extensible Properties

Python is one of the many programming languages that has portability and extensibility. With these two properties, one can cross-check the operations of Python consistently. Notably, the extensible ability of Python means it can be integrated with other languages. These languages include Java and even C and C++ because of their .Net components.

Data Science

A python is an essential tool that every data scientist must-have. The language supports many frameworks, including Django, Pyramid, and many more. With Python through data science, educationists can create the outputs they want using given inputs.

Students can quickly learn better ways of handling and processing data as well from their school days. These will prepare them ahead of time for the demand for data and information in the future.

Readable and Maintainable Code

In the educational environment, one of the uses of Python is coding. With Python’s syntax, you can quickly write specific concepts without having to write in other codes. Unlike most languages out there, code readability is essential.

Python’s syntax is just like the English language. However, the language allows you to use keywords instead of punctuation. With a readable and clean code base, you can take good care of software updates without too much work or wasted energy.

5 reasons why Python is a popular teaching language

The following five are the best reasons to use python in education:

  1. Code readability
  2. Obvious simplicity
  3. Open extensibility
  4. Cross-platform runability
  5. Humanity

Python Selenium Tutorial | Selenium Python Training for Beginners

Python is the most famous programming language that helps developers to code any logic for an application or game or website easily and quickly. The most powerful tool to control web browsers via programs and perform browser automation is ‘Selenium‘. This is helpful for all browsers and also operates on all major operating systems. The scripts of selenium are written in different languages ie., Python, Java, C#, etc.

Here, we will be working with Python. This BTech Geeks Python Selenium Tutorial includes all topics related to it like Selenium WebDriver, why learn selenium with python, How to integrate Selenium with Python Unit Tests. Moreover, it covers selenium with python concepts from basics to advanced and professional uses.

Python Selenium | Selenium with Python Tutorial PDF for Beginners

Related Articles:

What is Selenium?

Selenium is the most widely utilized open-source Web UI (User Interface) automation testing suite, that can be downloaded from the internet without spending anything. It can be efficiently deployed on platforms like Windows, Linux, Solaris, and Macintosh.

Selenium is a functional testing tool and also fits with non-functional testing tools as well. Currently, the Selenium Web driver is most familiar with Java and C#. The test scripts of Selenium can be coded in any of the supported programming languages and can be run straight away in most modern web browsers.

what is selenium

Browsers supported by Selenium

The list of most commonly supported browsers by the selenium are included:

  • Internet Explorer,
  • Mozilla Firefox,
  • Google Chrome, and
  • Safari.

Why learn Selenium Python?

  • Easier to understand and implement: The commands of selenium are classified in terms of various classes that make them easy to understand and implement.
  • Open Source and Portable: It is an open-source and portable web testing framework.
  • Cost reduction for Business Clients: By using the automation testing tool, the business can save paying the testers their salary. It saves time, cost benefits to the business.
  • Less burden and stress for testers: As we said earlier, the amount of time needed to do testing on every new build is lessen to zero, almost. Thus, the burden of the tester gets reduced.
  • Combination of tool and DSL: It is a blend of tools and Domain Specific Language (DSL) to perform different types of tests.

Why we use Selenium for automation testing?

The most famous automation testing tool is Selenium. Automation testing means the process of converting any manual test case into test scripts using automation tools like Selenium. If you want to know why we use Selenium for automation testing then make sure to have a look at the following aspects:

  • We can write a Selenium test script in any programming language such as Java, Python, C#, Ruby, Perl, as so on.
  • The test scripts can be performed in various OS platforms like Windows, Mac, Linux, etc.
  • Also, these scripts can be implemented on various browsers like Google Chrome, Mozilla Firefox, Internet Explorer, Safari, etc.
  • The testing tool of selenium can also be associated with various testing tools like TestNG, Junit for managing the test cases, and generating test reports.
  • To do endless testing, Selenium is combined with Maven, Jenkins, and Docker testing tools.

How to integrate Selenium with Python Unit Tests

In this section, we are going to discuss how to integrate Selenium tests with python unit tests and try to understand the concept easily. Thus, we will be using the unit test module in python.

import unittest
from selenium import webdriver
from selenium.webdriver.common.keys import Keys

class ChromeSearch(unittest.TestCase):

def setUp(self):
self.driver = webdriver.Chrome('./chromedriver')

def test_search_in_python_org(self):
driver = self.driver
self.assertIn("Python", driver.title)
elem = driver.find_element_by_name("q")
elem.send_keys("getting started with python")
assert "" == driver.current_url

def tearDown(self):

if __name__ == "__main__":

In this case, you require to set up the driver object when initializing the unit test class via the .Chrome() method. In the single test that we prove, the same text is put on the search bar and the resultant change in URL is compared to the URL that was seen first. You may additionally address a separate test for another browser and reuse the same functionality.

Why to choose Python over Java in Selenium

The important points that support python over java to work with selenium is as follows:

  • Java programs tend to run slower compared to Python programs.
  • Java uses static typing, while Python uses dynamic typing.
  • Python is easier and also extra compact compared to Java.
  • Java employs traditional braces to start and ends blocks, while Python uses indentation.

Selenium with Python

By using python, you can run a Selenium test script in two ways. They are as such:

  1. Configure Selenium using Python
  2. Configure PyDev in Eclipse.

Now, we will observe how we configure Selenium with the help of the Python programing language:

1. Configure Selenium using Python

By following the below steps, we can configure Selenium using Python efficiently:

  • Download and install Python on Windows
  • Install Selenium libraries in Python
  • Download and install PyCharm
  • Create a new project and write the Selenium test script
  • Run and validate the test scripts.

2. Configure PyDev in Eclipse

One more way to perform the Selenium test script using Python is by Configure PyDev in Eclipse. To make it possible, we have to follow these two steps mentioned below:

  • Install PyDev
  • Execute the Selenium test scripts using Python.

Install Selenium Libraries With Python

At the time of python installation, Selenium libraries are not installed by default. In order to perform if the libraries are already present in your python, you have to open a command prompt and navigate to the path where you have installed a python and type “pip list“. By using this command, it will list all the libraries currently available in your Python.

intall selenium libraries with python using pip

List of Python Selenium Interview Questions and Answers

The most commonly asked Interview Question and Answer in Selenium with Python suites for both Freshers and Experienced Candidates are listed here by BTech Geeks :

  1. What Is Python?
  2. What is Selenium?
  3. How Python Can Be Used In Software Testing?
  4. What Python Frameworks Do You Know?
  5.  What is the commands for downloading selenium?
  6. What Tools That Helps Python Development Do You Know?
  7. What Is The Difference Between “xrange” And “range”?
  8. What Is A “unit test” In Python?
  9. How can we implement unittest framework in our scripts?
  10. Define “module” And “package”?

What can you do with python: Usage guide | 9 Cool Things to do with Python Programming

What Can You Do With Python Usage Guide

Python Programming Language is a very familiar programming language among developers. It is simple and enjoyable to write scripts to automate and build stuff. A complete guide on what can you do with python is in front of your eyes. Kindly python learners and newbies are recommended to check out this python usage tutorial and learn what others do with python and what you do with python.

Along with this, you may also gain enough knowledge on how to learn python and what python programmers can make by using this easy and simple programming language. In case, you don’t have any idea how python is used in education then make a move and click on the link provided here itself.

This What Can You Do With Python Usage Guide will helo you learn the following stuff: 


  • You should know python’s basic syntax.
  • You should have Python installed on your machine.

What can you do with Python?

Python is a general-purpose language. You can use it with any kind of application. You can use it as a scripting language to solve some of your daily problems.

A few of the common use cases are as follows:

  • Creating bots
  • Scraping websites
  • Machine learning, data visualization, and analysis
  • Web Development with frameworks like Django and Flask
  • Game development with Pygame
  • Mobile apps with frameworks like Kivy

Scripting with Python

Let’s take an example. If you’re like photography, you probably have an issue with image naming. After a long day of shooting, you came back to your studio. You filtered the images and put the selected images in a directory. This directory contains different images with different names.

You need to make your work easier by naming all images in a specific format. Let’s say the image name will consist of the place name and sequence numbers. This makes it easy for you to search for images. By using Python you can create a simple script that will go through all files with type JPEG in this folder and rename it.
Scripting with PythonLet’s see an example:

import os 

def main():
    path = input("Enter folder path: ")
    folder = os.chdir(path)

    for count, filename in enumerate(os.listdir()): 
        dst ="NewYork" + str(count) + ".jpeg"
        os.rename(src, dst) 

if __name__ == '__main__':

As you can see Python is an efficient tool for automation. It helps drive your focus to important things.

What else you can do with Python? You can have an FTP server with python. Using the command

python3 -m http.server 8000

You can run a python server. The server will be accessible over your local network. You can open your browser from a different device. Type your network ip : 8000 . You will have access to all files in this folder. You can use it to transfer files from different devices in the same network.

Such a simple usage of Python in your daily life will make it easier. Talking about servers that leads us to

Python usage in web-based applications

Python is very popular in web development with frameworks like Flask and Django. If you want to build your own eCommerce store. You can use the Django framework. After installing it you will have a ready project structure to build your website. Let’s see how easy it is to install Django.

# checking django version 
$ python -m django --version

# if you don't have django installed you can install it using pip
$ python -m pip install Django

Note: if didn’t have pip installed you should follow our article for installing it.

After installing Django Now you can create your project.

$ django-admin startproject my-eCommerce-Website

You will have a project structure like below


To proceed with the Django project. You should install your project applications

$ python startapp store

Inside your Django project, you would have all your apps. Each app will have a different folder.


You can learn more about Django in their official documentation. You can that getting started with Python for web development is pretty easy.

Python has different usage in different fields Like Data science, game, and mobile development. we talked in detail about it in the previous article.

How to learn Python?

Learning Python is a continuous process. Some people prefer Courses, Others prefer hands-on experience. It really doesn’t matter the source you will learn from. You will need to apply what you learned to build useful Python applications. We’ve talked in our previous article about the various resources to learn it. check it here. If you already know it’s basic syntax. Your focus now should be on learning how to build python’s applications.

It depends on the type of application you want to build.  If you’re looking for building a web-based application. The recommendations will be Flask And Django.

Where you can learn Flask?

  • The Flask documentation has a get stating section. It’s very helpful to learn the basics of Flask.
  • Coursera Just released a new Course to learn Flask. It’ll guide you through all Flask details.
  • The tutorial point has a great organized Flask tutorial.

With all the above resources, you’ll build a Flask application. That will give you hands-on experience with Flask.

Where you can learn Django?

  • Django documentation is a detailed resource for learning Django. With its starting guide.
  • Django is the most popular Python framework. Coursera has created a full specialization to learn it.
  • The tutorial point has created an organized source to learn Django. It’s called the Django tutorial.

Again it doesn’t matter what is the source. You need to try writing code and build a project by yourself. It’ll help you build your portfolio.

If you’re looking for building machine learning projects. Udacity and Coursera are great resources to learn it.

Where you can learn Machine Learning?

  • Udacity has an introductory course on machine learning. It’ll guide you through machine learning and artificial intelligence.
  • Stanford has a great course too on Coursera to learn machine learning. It’s presented by Andrew Ng. He is one of the top in the Machine learning and AI fields.
  • If you’re prefer reading. The Tutorial point has a tutorial about Machine learning. It’ll give you the needed knowledge to know more about Machine learning.

What is the time to learn Python?

Learning Python as basics will not take a lot of time. It can be from two to four weeks to be comfortable with it. Learning other frameworks on top of Python might take time. It depends on the field and the difficulty of the framework. Learning how to build an application with Flask or Django might take 4 to 6 weeks. For Machine learning, it can take longer. Machine learning is a more complicated field. It takes more time to learn. Note that time can differ depends on your learning process. Keep in mind that fields Like Machine learning or Data science will require a mathematical background. This might make it longer to learn.

How Much Python programmers make?

Python is used in different industries. This makes it easy to find a job with it. In terms of Your salary, a lot of factors will control this.

  • Your experience
  • Your country
  • Company business size
  • Type of application you’re building

That’s why the more you learn more about it the more is always better. For Exact numbers about your expected salary that matches your experience. You should check Glassdoor. It’ll help you know the average number in your area based on skills.

9 Cool Things You Can Do Using Python

Here we are trying to include various domains that you can do great stuff with python. Have a look at the list of some interesting functionalities that you can do with Python:

  1. For Web Development
  2. Automation and Scripting
  3. Automating your daily routine tasks
  4. Automatically detects what’s in the image
  5. Web Scraping
  6. Data analysis, manipulation, and visualization
  7. Create games with Python
  8. Data Science and Machine Learning
  9. Building robotics applications


Python has a lot of useful usages. It is based on the type of application you want to build. Learning Python is an ongoing process. The more you learn about python the more applications you can build. It’ll help you get a job faster and increase your salary.

Stack Tutorial: an Implementation Beginner’s Guide


In order to understand this stack tutorial, you’ll need to learn the Stack data structure which requires the following:

  1. Python 3
  2. Basic data structure concepts like List (Click here to refresh List concepts)
  3. OOP concepts

What is a Stack?

Howdy! If you are reading this article, you are about to learn one of the very basic and most useful data structure concepts. If you know other languages like C or C++, implementation of stack might be tricky (since you will have to keep track of pointers!) but not with Python. Python is so amazing that you can use lists to easily implement them but you will also learn how to implement using pointers to adopt a language agnostic way. But first, let’s understand what they are. If you are already familiar with this, you can skip to the implementation section.

When you hear the word Stack, the first thing that comes to your mind may be a stack of books, and we can use this analogy to explain stacks easily! Some of the commonalities include:

  1. There is a book at the top of the stack (if there is only one book in the stack, then that will be considered the topmost book).
  2. Only when you remove the topmost book can you get access to the bottom ones. No Jenga games here! (Also assume that you can only lift one book at a time).
  3. Once you remove all the books from the top one by one, there will be none left and hence you cannot remove any more books.

Check out this fun game called Towers of Hanoi which beautifully demonstrates how a Stack works. Read the instructions carefully and turn off the music (it is awfully loud!).

What is a Stack

To describe the above points programmatically:

  1. Keep track of the topmost element as this will give you the information about the number of elements in the stack and whether the stack is empty/full (if the stack is empty then top will be set to 0 or a negative number)
  2. The last element to enter the stack will always be the first to leave (Last In First Out – LIFO)
  3. If all the elements are removed, then the stack is empty and if you try to remove elements from an empty stack, a warning or an error message is thrown.
  4. If the stack has reached its maximum limit and you try to add more elements, a warning or error message is thrown.

Things to remember:

  1. The entry and exit of elements happens only from one end of the stack (top)
  2. Push – Adding an element to the Stack
  3. Pop – Removing an element from the Stack
  4. Random access is not allowed – you cannot add or remove an element from the middle.

Note: Always keep track of the Top. This tells us the status of the stack.

How to implement Stack?

Now that you know what a Stack is, let’s get started with the implementation!

Stack implementation using List

Here we are going to define a class Stack and add methods to perform the below operations:

  1. Push elements into a Stack
  2. Pop elements from a Stack and issue a warning if it’s empty
  3. Get the size of the Stack
  4. Print all the elements of the Stack
class Stack:

    #Constructor creates a list
    def __init__(self):
        self.stack = list()

    #Adding elements to stack
    def push(self,data):
        #Checking to avoid duplicate entries
        if data not in self.stack:
            return True
        return False

    #Removing last element from the stack
    def pop(self):
        if len(self.stack)<=0:
            return ("Stack Empty!")
        return self.stack.pop()
    #Getting the size of the stack
    def size(self):
        return len(self.stack)

myStack = Stack()
print(myStack.push(5)) #prints True
print(myStack.push(6)) #prints True
print(myStack.push(9)) #prints True
print(myStack.push(5)) #prints False since 5 is there
print(myStack.push(3)) #prints True
print(myStack.size())  #prints 4 
print(myStack.pop())   #prints 3
print(myStack.pop())   #prints 9
print(myStack.pop())   #prints 6
print(myStack.pop())   #prints 5
print(myStack.size())  #prints 0
print(myStack.pop())   #prints Stack Empty!

NOTE: We are not worried about the size of the stack since it is represented by a list which can dynamically change its size.

Stack implementation using Array

Python Lists have made it so easy to implement Stack. However, if you want to implement Stack language agnostically, you have to assume that lists are like arrays (fixed in size) and use a Top pointer to keep a tab on the status of the stack. Check this animation to understand how it works.


  1. Declare a list and an integer MaxSize, denoting the maximum size of the Stack
  2. Top is initially set to 0
  3. Push operation:
    1. Check if Top is less than the MaxSize of the Stack
      1. If yes, append data to stack and increment top by 1
      2. If no, print stack full message
  4. Pop operation:
    1. Check if Top is greater than 0:
      1. If yes, pop the last element from the list and decrement top by 1
      2. If no, print stack empty message
  5. Size operation:
    1. The value of the Top pointer is the size of the Stack


class Stack:
    def __init__(self):
        self.stack = list()
        self.maxSize = 8 = 0
    #Adds element to the Stack
    def push(self,data):
            return ("Stack Full!")
        self.stack.append(data) += 1
        return True
    #Removes element from the stack
    def pop(self):
            return ("Stack Empty!")
        item = self.stack.pop() -= 1
        return item
    #Size of the stack
    def size(self):

s = Stack()
print(s.push(1))#prints True
print(s.push(2))#prints True
print(s.push(3))#prints True
print(s.push(4))#prints True
print(s.push(5))#prints True
print(s.push(6))#prints True
print(s.push(7))#prints True
print(s.push(8))#prints True
print(s.push(9))#prints Stack Full!
print(s.size())#prints 8        
print(s.pop())#prints 8
print(s.pop())#prints 7
print(s.pop())#prints 6
print(s.pop())#prints 5
print(s.pop())#prints 4
print(s.pop())#prints 3
print(s.pop())#prints 2
print(s.pop())#prints 1
print(s.pop())#prints Stack Empty!

Note: Element 9 was not added and hence size remains 8.

Apart from the methods described above, you can add methods to return the top element, check if the stack is empty etc.


One of the main application of Stacks is in recursion. Be sure to check this tutorial to know what recursion is.  Once you are familiar with this concept, try to solve these puzzles using recursion – sentence reversal and balancing brackets. Happy Pythoning!