How to Pass Python Variables to Javascript

Estimated reading time: 4 minutes

In our recent blog posting How to Pass a Javascript Variable to Python using JSON, we demonstrated how to easily use AJAX to pass whatever data you wanted and then manipulate it with Python.

In this blog posting, we are going to show how to do this the other way around. The scenario is that you have an application and or website that wants to use data generated through Python, but let Javascript then use it within the application.

As Python can be connected to numerous databases and files ( txt, excel) etc, this piece of logic is very useful for the programmer looking to integrate both programming languages.

Let’s start looking at the code, and see how this can be achieved.

Step 1 – What Files are generated?

This program uses Python Flask to create a web page, that has a drop-down menu. The two files used to generate this are as follows:

(A) app.py – This is the python file that creates a website and loads a template HTML file as outlined below.

(B) Index.html – This is the template file that loads into the browser and runs all the javascript. The javascript loaded here also loads the python data passed over from app.py

Step 2 – APP.PY code overview

The Python library that enables webpage creation is called Flask, and as can be seen below it has to be imported.

In addition, we need to also import render_template which tells the program to go to the templates folder and load “Index.HTML”

The variable that is been passed to JavaScript is called name, and these are the values that you will see in the web browser when it is loaded.

from flask import Flask, render_template

app = Flask(__name__)

@app.route('/')
def index():
    name = ['Joe','John','Jim','Paul','Niall','Tom']
    return render_template('index.html', name=name)

if __name__ == "__main__":
    app.run(debug=True)

Step 3 – Index.HTML overview

Here is the template HTML file that runs in the browser. You can add CSS etc to this to make it look nicer and more user friendly.

As you can see it has the usual HTML tags appear as part of a website.

Well look at some of the code further:

In this bit <select id =’select’> </select>, this is the dropdown menu that will appear when Index.html is opened. It will store all the values passed from python. Note that its id is “select”, this will be used later on.

The main parts to focus on next is between <script></script>. This is what reads in the python data and populates it to the dropdown menu.

In step 2 we mentioned that there was a variable called “name”, with values to be passed over.

This is acheived on this line:

var select = document.getElementById(“select”), test = {{ name | tojson }};

Notice that name appears here, and this is referencing back to the exact same value that was discussed in step 2.

For the rest of the lines, I have explained with comments what each does.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Pass Python variable to Javascript</title>
</head>
<body>

<select id ='selectvalue'>
</select>
<script>
    //name = ['Joe','John','Jim','Paul','Niall','Tom']

    var selectvalue = document.getElementById("selectvalue"), test = {{ name | tojson }};
    //The increment operator (++) increments (adds one to) its operand and returns a value.
    for(var i = 0; i < test.length; i++) // This line checks for the length of the data you feeding in i.e the no of items
         {
var selection = document.createElement("OPTION"), // This line creates a variable to store the different values fed in from the JSON object "TEST"
txt = document.createTextNode(test[i]); // This just reads each value from the test JSON variable above
selection.appendChild(txt); // This line appends each value as it is read.
selection.setAttribute("value",test[i]); // This line sets each value read in as a value for the drop down
selectvalue.insertBefore(selection,selectvalue.lastChild); //This reads eah value into the dropdown based on the order in the "TEST" above.
 }
</script>
</body>
</html>

Step 4 – What the output looks like!

From step 2, these are values we asked to be used in Javascript to populate a dropdown:

name = [‘Joe’,’John’,’Jim’,’Paul’,’Niall’,’Tom’]

Python variable passed to Javascript

How to pass multiple lists to a function and compare

Estimated reading time: 3 minutes

Have you ever been faced with a situation where you have multiple lists you want to compare, and need a function that will read them in and show you what is different between them all?

In this scenario, this can be handy where you have large data sets and need a quick way to find those differences and fix them where appropriate.

Comparing two lists

Let’s look and see what is going on below.

First of all, we have defined two lists. The only difference between the two is that one has a value of six, the other does not.

Next, we have the function “comparelists”. What this is doing is taking in the two lists as parameters (a,b), and then processing them.

The lists are passed as arguments to the parameters in this line ===> comparelists(list1,list2)

The parameters a are assigned to list1, and b is assigned to list2.

The main bit of the function is the list comprehension, and it is doing the following:

  1. x for x is basically creating a variable x, and starting a loop that goes through all the values of b.
  2. Each iteration of x is stored and compared with a.
  3. “If x not in a” completes the comparison and if true returns the value, otherwise moves to the next value.

As a result of this logic, it can be seen that six is the only value returned, and this is what we are expecting based on a visual inspection.

#Reading in two lists
list1 = [1,2,3,4,5] # parameter a below
list2 = [1,2,3,4,5,6] # parameter b below

def comparelists(a,b):
    z = [x for x in b if x not in a] #list comprehension
    print(z)

comparelists(list1,list2)

Output:

[6]

Comparing more than two lists

In the logic above, we saw that two lists can be compared. But what if you want to compare one list against two other lists?

That is easily done, with some tweaking to the code.

As before the three lists are defined, created as arguments in comparelists(list1,list2,list3), and then passed to the function parameters a,b,c.

The only difference in this logic is that the list comprehension is written a little different as follows:

  1. x for x is basically creating a variable x, and starting a loop that goes through all the values of b. ====> This is the same as above
  2. Each iteration of x is stored and compared with a and c ===> This line is different as comparing to two lists now.
  3. “if x not in a and x not in c” ====> Here we have two comparisons
    • Checking for the value x from b in a
    • Checking for the value x from b in c
  4. The value of six is correct as it is not in either a or c.
  5. Note the if statement is specific as to what to check for with the “and” statement. This can be changed to suit your needs.
list1 = [1,2,3,4,5] # parameter a below
list2 = [1,2,3,4,5,6] # parameter b below
list3 = [1,2,3,7,8,9] # parameter c below

def comparelists(a,b,c):
    z = [x for x in b
         if x not in a and x not in c] #list comprehension
    print(z)

comparelists(list1,list2,list3)

Output:
[6]

What is an array in Python?

Estimated reading time: 3 minutes

Python arrays are used to manage how data is analysed and have the data in a structured form for the data analyst or data scientist to use.

What is an Array?

An array has the following properties:

  1. The data in it is of the same data type.
  2. The data is stored as contiguous memory locations, meaning meaning the lowest value is at index 0 and the last value is at the highest index value.
  3. Arrays assign index values to each data point contained within it.
  4. You can append values to the array.
  5. You can delete values from the array, that is it is mutable.

What are the differences between arrays and lists?

For the most part arrays and lists are the same, but with one difference:

A list can store any data type you want e.g. strings, integers etc.

On the other hand, arrays can only store data that is of the same data type.

What are the different ways that I can create an array?

(A) Use Numpy

import numpy as np

a = np.array([1,2,3])

print(a)
print(type(a))

Output:
[1 2 3]
<class 'numpy.ndarray'>

(B) Use array

import array as test_array

a = test_array.array('i',[1,2,3])

print(a)

Output:
array('i', [1, 2, 3])
<class 'array.array'>

On the Python.org website, below are the list of values that can be populated into the above program, depending on what your need is:

When should I use arrays?

It really depends on the nature of your python program, but below are some examples that may help you make a decision:

(A) Many variables of the same type : There maybe a scenario where you have to create an array to store data that is of the same data type. For example you have a list of codes to look up against, which are all integers.

(B) Faster and more efficient : If speed is what you are looking for using arrays, will help improve performance of your computer program, using lists is much slower.

(C) Compactability and efficiency : If the nature of your program needs to store large amounts of data that needs to be accessed quickly , then this would be a good reason to use them.

(D) Ability to retrieve data quickly through indexing: As arrays have index values associated with them, they data can be easiy retrieved.

(E) Need to compute some mathematical values: Arrays are excellent for any numerical operations you need to complete, as the level of coding is minimal.

import array as test_array

a = test_array.array('i',[1,2,3])

mydivider = 2

mynewlist = [x / mydivider for x in a]
print(mynewlist)

result:
[0.5, 1.0, 1.5]

So in summary:

Speed , efficiency and ease of use are the main reasons to use an array.

We use arrays here in how to show percentage differences between files in python , why not go over and see it in action!

How to Compare Column Headers in CSV to a List in Python

Estimated reading time: 3 minutes

So you have numerous different automation projects in Python. In order to ensure a clean and smooth straight-through processing, checks need to be made to ensure what was received is in the right format.

Most but not all files used in an automated process will be in the CSV format. It is important there that the column headers in these files are correct so you can process the file correctly.

This ensures a rigorous process that has errors limited.

How to compare the headers

The first step would be to load the data into a Pandas data frame:

import pandas as pd

df = pd.read_csv("csv_import.csv") #===> Include the headers
print(df)

The actual original file is as follows:

Next we need to make sure that we have a list that we can compare to:

header_list = ['Name','Address_1','Address_2','Address_3','Address_4','City','Country']

The next step will allow us to save the headers imported in the file to a variable:

import_headers = df.axes[1] #==> 1 is to identify columns
print(import_headers)

Note that the axis chosen was 1, and this is what Python recognises as the column axes.

Finally we will apply a loop as follows:

a = [i for i in import_headers if i not in header_list]
print(a)

In this loop, the variable “a” is taking the value “i” which represents each value in the import_headers variable and through a loop checks each one against the header_list to see if it is in it.

It then prints out the values not found.

Pulling this all together gives:

import pandas as pd

df = pd.read_csv("csv_import.csv") #===> Include the headers
print(df)

#Expected values to receive in CSV file
header_list = ['Name','Address_1','Address_2','Address_3','Address_4','City','Country']

import_headers = df.axes[1] #==> 1 is to identify columns
print(import_headers)


a = [i for i in import_headers if i not in header_list]
print(a)

Resulting in the following output:

As can be seen the addresses below where found not to be valid, as they where not contained within our check list “header_list”

IndexError: list index out of range

Estimated reading time: 3 minutes

Are you working with lists and getting the error IndexError: list index out of range while using Python? There is a very simple explanation to this, and its fix is very easy.

First of all lets understand what is going on with the list.

Traceback (most recent call last):
  File "C:/Users/haugh/OneDrive/dataanalyticsireland/YOUTUBE/IndexError_list_index_out_of_range/INDEX_ERROR_LIST_INDEX_OUT_OF_RANGE.py", line 4, in <module>
    print(data[4])
IndexError: list index out of range

Lists and their index values

In the below list, we have outputted its values and index values.

data = ['a','b','c','d']
for (i,item) in enumerate(data, start=0): #===> Loops through list and applies index values starting at zero
    print(i,item)

Output:
0 a
1 b
2 c
3 d

Process finished with exit code 0

As can be seen the program returns the list values and their indexes. Note that the index starts at zero as we have set start=0.

Start=0 can be set to any value you like, as can be seen here:

data = ['a','b','c','d']
for (i,item) in enumerate(data, start=1): #===> Loops through list and applies index values starting at zero
    print(i,item)

Output:
1 a
2 b
3 c
4 d

Process finished with exit code 0

OR

data = ['a','b','c','d']
for (i,item) in enumerate(data, start=22): #===> Loops through list and applies index values starting at zero
    print(i,item)

Output:
22 a
23 b
24 c
25 d

Process finished with exit code 0

The purpose of the index value is to tell the program where to start its index from, if left empty it starts at zero.

Lists and the no of index values

In the above examples the index values all occur on four rows.

This is important as when you are looping through the rows, it will not go beyond the length of the rows.

So in this example the enumerate function specifically counts the no rows and stores the index values with each, and then loops through the list till it hits the last one, without error.

data = ['a','b','c','d']
for (i,item) in enumerate(data, start=0): #===> Loops through list and applies index values starting at zero
    print(i,item)

Output:
0 a
1 b
2 c
3 d

Process finished with exit code 0

How to fix the error IndexError: list index out of range

So the reason we get the below is that the line print(data[4]) is looking for the row with index value 4, but we know that from observation that does not exist.

To fix this we would change the value 4 in print(data[4]) to any of 0,1,2,3, as they are the index values associated with the list.

data = ['a','b','c','d']
for (i,item) in enumerate(data, start=0): #===> Loops through list and applies index values starting at zero
    print(i,item)
print(data[4])

Output:
Traceback (most recent call last):
  File "C:/Users/haugh/OneDrive/dataanalyticsireland/YOUTUBE/IndexError_list_index_out_of_range/INDEX_ERROR_LIST_INDEX_OUT_OF_RANGE.py", line 4, in <module>
    print(data[4])
IndexError: list index out of range
0 a
1 b
2 c
3 d

Applying a correct valid index value:
data = ['a','b','c','d']
for (i,item) in enumerate(data, start=0): #===> Loops through list and applies index values starting at zero
    print(i,item)
print(data[3])

Yields with no error:
0 a
1 b
2 c
3 d
d

So in summary when working with lists and their index values it is important:

(A) Understand the length of your list.

(B) Where your index values start and finish.

This error is easily fixable, but in your code you just need to make sure that you referencing values that are in the range of your index values.

YouTube channel lists – Python Data Cleansing

Ever had a process where you received a set of data, and it took a bit of effort to cleanse the data, so it looks the way you want it?

The world of data processing and data exchanged between servers and organisations needs careful attention, one person’s idea of clean data might not be other persons, hence the difference between the two can lead to data issues.

Experian had an excellent article What is data cleansing? in that they talk about several factors about data:

  • It could be incorrect
  • and incomplete
  •  and Duplicated

 

One of the things they highlighted also is that under GDPR, organisations have to deal with more focus on data been accurate, complete and up to date.

We are putting together several videos in this area over time, so you will start to see them as they go up.

Please like and share through the social media buttons shared on the page here, thanks for watching!

Data Analytics Ireland

 

 

 

Python Tutorial: How to sort lists

Estimated reading time: 2 minutes

Following on from our post on how to use Python lists have you ever wondered how to sort lists for your Python project?

Our latest video on lists will go through some of the techniques available so that you can get an idea of how to structure your data and sort.

Getting to understand how to implement

In this latest video we will look at:

  • sort() method
  • sorted() function
  • sorting a list through a function

 

Adding in those extra bits to help make the process smoother

Have you thought about sorting ascending/descending?

  • There is also a discussion on this topic as well, and while an index is available for the list, which you may feel does not merit sorting, there could be other logical reasons to implement sorting.
  • Leaving out the reverse = True/False in the sorted method can have an impact, though if you require it left out of the list you have created, automatic ascending will be the default.

On this channel, we have discussed a number of different ways to manage your data. In thinking about sorting a list, why would you want to do this?

Some common reasons are:

  • To visually see if there are duplicates, either on the screen or printed out.
  • If other objects are dependant on the list, say a combo box, then having duplicates visible can help to reduce the size of their contents.
  • Iteration – If you are looking to iterate over a list, it will be quicker if it is sorted.

If you want to learn about lists, using them, and how how they can be iterated over, why not visit Data Analytics Ireland YouTube channel, there are lots of videos there that will help explain the concepts discussed here further.

To get some more links on this topic click here python sort method, it is a blog posting from our website that has some useful links and explanations for you.

YouTube channel lists – Python Lists

Estimated reading time: 2 minutes

Python lists are used extensively in projects, as a result it is important to understand their structure.

Some of the things they can be used for:

  1. Lookup values for comparisons.
  2. Passing data to them to store to be referenced elsewhere.
  3. As part of a loop, store values that have been found through the loop logic.

With methods are associated with lists?

  1. Append – Add values to the end of the list
  2. Extend – adds values from an iterable object to the end of the list.
  3. Insert – You can insert an item to a certain position in a list.
  4. Remove – Remove the first value in a list that has a value that was asked to be looked for.
  5. Pop – This also removes a value at a certain position and returns, consequently if no position is specified then it removes the last item and returns it.
  6. Clear – Removes all items from the list.
  7. Index – returns the index value of the first item found that was asked to be searched for.
  8. Count – returns the number of times an item that was searched or was found in a list.
  9. Sort – Sorts the items in the list.
  10. Reverse – This reverses the items in the list.
  11. Copy – This makes a copy of the list.

What are the properties of a list?

The data type has the following attributes, that make it really useful for a vast array of scenarios:

  • They are ordered – Whatever order the list is a unique characteristic of the list, furthermore changing the order makes it a different list.
  • You can use their index to access the value.
  • They are mutable, meaning you can apply any of the above methods on them.
  • They can contain strings, integers etc, accordingly, there is no restriction on what can be in the list.

Check out the below video playlist from our YouTube channel, they will help explain more about lists:

On this website you can also read about how to compare two lists in Python or how to sort lists using rstudio in addition to this blog post.

We hope you enjoy it!

Data Analytics Ireland