How to Pass Python Variables to Javascript

Estimated reading time: 4 minutes

In our recent blog posting How to Pass a Javascript Variable to Python using JSON, we demonstrated how to easily use AJAX to pass whatever data you wanted and then manipulate it with Python.

In this blog posting, we are going to show how to do this the other way around. The scenario is that you have an application and or website that wants to use data generated through Python, but let Javascript then use it within the application.

As Python can be connected to numerous databases and files ( txt, excel) etc, this piece of logic is very useful for the programmer looking to integrate both programming languages.

Let’s start looking at the code, and see how this can be achieved.

Step 1 – What Files are generated?

This program uses Python Flask to create a web page, that has a drop-down menu. The two files used to generate this are as follows:

(A) app.py – This is the python file that creates a website and loads a template HTML file as outlined below.

(B) Index.html – This is the template file that loads into the browser and runs all the javascript. The javascript loaded here also loads the python data passed over from app.py

Step 2 – APP.PY code overview

The Python library that enables webpage creation is called Flask, and as can be seen below it has to be imported.

In addition, we need to also import render_template which tells the program to go to the templates folder and load “Index.HTML”

The variable that is been passed to JavaScript is called name, and these are the values that you will see in the web browser when it is loaded.

from flask import Flask, render_template

app = Flask(__name__)

@app.route('/')
def index():
    name = ['Joe','John','Jim','Paul','Niall','Tom']
    return render_template('index.html', name=name)

if __name__ == "__main__":
    app.run(debug=True)

Step 3 – Index.HTML overview

Here is the template HTML file that runs in the browser. You can add CSS etc to this to make it look nicer and more user friendly.

As you can see it has the usual HTML tags appear as part of a website.

Well look at some of the code further:

In this bit <select id =’select’> </select>, this is the dropdown menu that will appear when Index.html is opened. It will store all the values passed from python. Note that its id is “select”, this will be used later on.

The main parts to focus on next is between <script></script>. This is what reads in the python data and populates it to the dropdown menu.

In step 2 we mentioned that there was a variable called “name”, with values to be passed over.

This is acheived on this line:

var select = document.getElementById(“select”), test = {{ name | tojson }};

Notice that name appears here, and this is referencing back to the exact same value that was discussed in step 2.

For the rest of the lines, I have explained with comments what each does.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Pass Python variable to Javascript</title>
</head>
<body>

<select id ='selectvalue'>
</select>
<script>
    //name = ['Joe','John','Jim','Paul','Niall','Tom']

    var selectvalue = document.getElementById("selectvalue"), test = {{ name | tojson }};
    //The increment operator (++) increments (adds one to) its operand and returns a value.
    for(var i = 0; i < test.length; i++) // This line checks for the length of the data you feeding in i.e the no of items
         {
var selection = document.createElement("OPTION"), // This line creates a variable to store the different values fed in from the JSON object "TEST"
txt = document.createTextNode(test[i]); // This just reads each value from the test JSON variable above
selection.appendChild(txt); // This line appends each value as it is read.
selection.setAttribute("value",test[i]); // This line sets each value read in as a value for the drop down
selectvalue.insertBefore(selection,selectvalue.lastChild); //This reads eah value into the dropdown based on the order in the "TEST" above.
 }
</script>
</body>
</html>

Step 4 – What the output looks like!

From step 2, these are values we asked to be used in Javascript to populate a dropdown:

name = [‘Joe’,’John’,’Jim’,’Paul’,’Niall’,’Tom’]

Python variable passed to Javascript

What is an array in Python?

Estimated reading time: 3 minutes

Python arrays are used to manage how data is analysed and have the data in a structured form for the data analyst or data scientist to use.

What is an Array?

An array has the following properties:

  1. The data in it is of the same data type.
  2. The data is stored as contiguous memory locations, meaning meaning the lowest value is at index 0 and the last value is at the highest index value.
  3. Arrays assign index values to each data point contained within it.
  4. You can append values to the array.
  5. You can delete values from the array, that is it is mutable.

What are the differences between arrays and lists?

For the most part arrays and lists are the same, but with one difference:

A list can store any data type you want e.g. strings, integers etc.

On the other hand, arrays can only store data that is of the same data type.

What are the different ways that I can create an array?

(A) Use Numpy

import numpy as np

a = np.array([1,2,3])

print(a)
print(type(a))

Output:
[1 2 3]
<class 'numpy.ndarray'>

(B) Use array

import array as test_array

a = test_array.array('i',[1,2,3])

print(a)

Output:
array('i', [1, 2, 3])
<class 'array.array'>

On the Python.org website, below are the list of values that can be populated into the above program, depending on what your need is:

When should I use arrays?

It really depends on the nature of your python program, but below are some examples that may help you make a decision:

(A) Many variables of the same type : There maybe a scenario where you have to create an array to store data that is of the same data type. For example you have a list of codes to look up against, which are all integers.

(B) Faster and more efficient : If speed is what you are looking for using arrays, will help improve performance of your computer program, using lists is much slower.

(C) Compactability and efficiency : If the nature of your program needs to store large amounts of data that needs to be accessed quickly , then this would be a good reason to use them.

(D) Ability to retrieve data quickly through indexing: As arrays have index values associated with them, they data can be easiy retrieved.

(E) Need to compute some mathematical values: Arrays are excellent for any numerical operations you need to complete, as the level of coding is minimal.

import array as test_array

a = test_array.array('i',[1,2,3])

mydivider = 2

mynewlist = [x / mydivider for x in a]
print(mynewlist)

result:
[0.5, 1.0, 1.5]

So in summary:

Speed , efficiency and ease of use are the main reasons to use an array.

We use arrays here in how to show percentage differences between files in python , why not go over and see it in action!

How to Create an XML file from Excel using Python

Estimated reading time: 3 minutes

Are you working on a data analytics project where you need to feed your data to a location that is able to process an XML file?

The ability to get your data into a structured format like XML has many benefits:

(A) You can transfer the data to a web service for processing.

(B) Multiple different formats of your raw data can be standardised, enabling quick conversion and processing.

(C) XML files can be read by multiple different programs, as long as you deliver them in the correct format.

(D) The receiver of data can easily read the XML file and store it on their database.

The ability to use this method to read and transfer data is a very powerful way to help a data analyst process large data sets.

In fact, if you are using cloud-based applications to analyse the information you are storing, this will quickly enable you to deliver the data.

What packages do I need in Python?

The first step is to import the following:

import pandas as pd
from lxml import etree as et

Next we want to read in the source data

In this instance, we are reading an excel file

raw_data = pd.read_excel(r'Link to where your data is stored including the full file name')

Now we want to start building the XML structure

The FIRST STEP is to define the root

root = et.Element('root')

The root is the parent of all the data items (tags) contained in the XML file and is needed as part of the structure

The SECOND STEP is to define the tag names that will store each row of the source data

for row in raw_data.iterrows(): ==> This is a loop that takes runs through each record and populates for each tag.
    root_tags = et.SubElement(root, 'ExportData') #=== > Root name
# These are the tag names for each row (SECTION 1)
    Column_heading_1 = et.SubElement(root_tags, 'Name')
    Column_heading_2 = et.SubElement(root_tags, 'Area')
    Column_heading_3 = et.SubElement(root_tags, 'NoPurchases')
    Column_heading_4 = et.SubElement(root_tags, 'Active')

###These are the values that will be populated for each row above
# The values inside the [] are the raw file column headings.(SECTION 2)
    Column_heading_1.text = str(row[1]['Name'])
    Column_heading_2.text = str(row[1]['Area'])
    Column_heading_3.text = str(row[1]['No Purchases'])
    Column_heading_4.text = str(row[1]['Active'])

The raw file looks like this:

NameAreaNo PurchasesActive
JohnDublin2Y
MaryGalway3N
JoeLimerick4N
JimmyKilkenny55Y
JenniferBelfast6N
SusanWaterford3Y
JakeCork1Y
BobbyDundalk11N
SarahSligo9N
CianEnnis8Y
Raw file data that will be imported into the XML file

The THIRD STEP is to create the XML file and populate it with the data from the source file

# This Section outputs the data to an xml file
# Unless you tell it otherwise it saves it to the same folder as the script.
tree = et.ElementTree(root) ==> The variable tree is to hold all the values of "root"
et.indent(tree, space="\t", level=0) ===> This just formats in a way that the XML is readable
tree.write('output.xml', encoding="utf-8") ==> The data is saved to an XML file

The XML output should look like the below

<root>
	<ExportData>
		<Name>John</Name>
		<Area>Dublin</Area>
		<NoPurchases>2</NoPurchases>
		<Active>Y</Active>
	</ExportData>
	<ExportData>
		<Name>Mary</Name>
		<Area>Galway</Area>
		<NoPurchases>3</NoPurchases>
		<Active>N</Active>
	</ExportData>
	<ExportData>
		<Name>Joe</Name>
		<Area>Limerick</Area>
		<NoPurchases>4</NoPurchases>
		<Active>N</Active>
	</ExportData>
	<ExportData>
		<Name>Jimmy</Name>
		<Area>Kilkenny</Area>
		<NoPurchases>55</NoPurchases>
		<Active>Y</Active>
	</ExportData>
	<ExportData>
		<Name>Jennifer</Name>
		<Area>Belfast</Area>
		<NoPurchases>6</NoPurchases>
		<Active>N</Active>
	</ExportData>
	<ExportData>
		<Name>Susan</Name>
		<Area>Waterford</Area>
		<NoPurchases>3</NoPurchases>
		<Active>Y</Active>
	</ExportData>
	<ExportData>
		<Name>Jake</Name>
		<Area>Cork</Area>
		<NoPurchases>1</NoPurchases>
		<Active>Y</Active>
	</ExportData>
	<ExportData>
		<Name>Bobby</Name>
		<Area>Dundalk</Area>
		<NoPurchases>11</NoPurchases>
		<Active>N</Active>
	</ExportData>
	<ExportData>
		<Name>Sarah</Name>
		<Area>Sligo</Area>
		<NoPurchases>9</NoPurchases>
		<Active>N</Active>
	</ExportData>
	<ExportData>
		<Name>Cian</Name>
		<Area>Ennis</Area>
		<NoPurchases>8</NoPurchases>
		<Active>Y</Active>
	</ExportData>
</root>

Additional XML data can be added

  1. Add more rows – All you need to do is add onto the source file and save. When you rerun the logic it will read in the extra information.
  2. Add more columns – All you need to do is go to the second step above add in a tag name to SECTION 1. Seperately you will need to add an additional column with data to the source file, and then add that column name to SECTION 2 as well

Python Tutorial: Pandas groupby columns ( video 2)

Pandas groupby using column values

In this second video how to groupby using pandas and as part of expanding the data analytics information of this website, we are looking to explain how you can use a groupby selection but only using the column values and not the column names.

Below we import our data into a dataframe, and then group as follows:

  • Aggregate function
  • Using the cut function and assigning values to bins.
  • Assigning labels to the data frame output based on the bin values.

 

 

Why would you want to use Pandas groupby and column values?

This video looks to help understand the why going by values might be easier than column names:

  • Column names can change from project to project, using by values allows easy implementation of getting the output regardless of the names used.
  • You could apply this to any Python class, and as long as you can inherit will allow the code to run smoothly.
  • Implementing by value allows a clear understanding of the desired output as the values are clearly understood to generate what is required.
  • You need to understand how data within your data set falls within a particular cohort:
    • This use of values in different programs just needs to change, the underlying logic remains the same.
    • Using column names still means that to group them, the logic still needs to be written.

Python tutorial: Pandas groupby ( Video 1)

In this first video about pandas groupby and as part of expanding the data analytics information of this website, we are looking to explain how you can use a groupby selection to sort your data into similar datasets better so they can be better analysed. In the video below, we import our data into a dataframe, and then group as follows:

  • Directly naming the column
  • Through get_group
  • Using a loop
  • Utilising a lambda function

 

 

Python tutorial: Create an input box in Tkinter

Using an tkinter input box for your data projects

There may be an occasion as you are building out a data science or data analytics project, checks need to be performed on the dataset as follows:

  •  Big data sets and speed requirements in conjunction with
  • The need to reduce the volume of data returned which is impeding performance

and this is where input boxes and Tkinter can help!

In the below video, we are demonstrating an introduction to using an input box and validating the input.

We demonstrate how to validate the data entered into the tkinter input box and return a message, this will ensure the user gets the correct data.

Types of uses for a tkinter input box are varied, here are some thoughts:

  • Use an input box to return a set of data for a particular day.
  • Using them to filter down the results to a particular cohort of data.
  • Conduct a string search to find data quality issues to be fixed.

Python tutorial: How to create a graphical user interface in Tkinter

How would you like to present your data analytics work better?

When starting your data analytics projects, one of the critical considerations is how to present your results quickly and understandably?

Undoubtedly this is true if you are only going to look at the results yourself.

If the work you do is a repeatable process, a more robust longer-term solution needs to be applied, this is where Tkinter can help, which is a python graphical user interface.

When you are importing tkinter, some of the functionality that can be used include:

  • Use them to build calculators.
  • They can show graphs and bar charts.
  • Show graphics on a screen.
  • Validate user input, through building entry widgets.

Where this all fits in with data analytics?

While going through a set of data and getting some meaning to it can be challenging, using the python graphical user interface tutorial below can help build the screens that will allow a repeatable process to display in a meaningful way.

Using the tkinter widget could help achieve the following:

  • Build a screen that shows data analytics errors in a data set, e.g. The number of blank column values in a dataset.
  • Another application is to run your analytics to show the results on a screen that can be printed or exported.
  • Similarly, you could also have a screen where a user selects several parameters that are fed into the data analytics code and produces information for the user to analyse.

There are many more ways that you could do this, but one of the most important things is that data analytics can be built into a windows environment using Tkinter.

These GUI applications are what the user would be used to currently seeing. As a result, this could help to distribute a solution across an enterprise to lots of different users.

Also, another benefit is that they will work on many different operating systems.

The only thing that needs to happen is that the requirements the user needs are defined, and the developer then builds on those, with the data analytics code run in the background of this program with Tkinter and output into a user-friendly screen for review.

How to create a combobox in tkinter

Estimated reading time: 2 minutes

Here we have delivered a complimentary video on How to create a graphical user interface in Tkinter .

It will demonstrate how a Combobox can be used to select values and then validate the entry chosen.

Using a Combobox in the computer programming world has been around for some time.

It is a useful way to select from a choice and could in many ways in data analytics help as the following examples show:

  • Select a date to filter a data set down to values that are in the dataset.
  • Using matplotlib to plot data points in charts, you could have dynamic values that change the diagram based on values chosen from the Combobox.
  • Utilizing data analytics reports that the user accesses, the Combobox could be used to change the data shown dynamically to allow comparisons.
  • When looking to fix data quality issues, use the Combobox to select values for a date that needs to be fixed, apply the fixes on screen, and then save back to the database.

Developing a Tkinter GUI and the possibilities it brings

In this video, we use ttk, written to help split the behavior of code from the code implementing its appearance.

You can see plenty more on it here ttk information. This is a handy piece of functionality as styling an object can interfere with how it works.

We also have a function that helps with the validation. In the below, it accomplishes the following:

  • Allows the combobox value selected to be retrieved.
  • Validates the entry chosen in the combobox using an if statement.
def checkifireland ():
    x = combolist.get() # asssigns the value inside the combobox to x so it can be processed
    if x == "Ireland":
        messagebox.showinfo("Correct answer", "You will love it in Ireland")
    else:
        messagebox.showinfo("Incorrect answer", "You should visit Ireland first!")

The effectiveness is especially handy as it helps to ensure that the code returned from the Combobox to the function is correct

The below video will take through this step by step and explain the concepts discussed above.

The next steps

There are many informative Python – working with excel videos that are on our YouTube channel.

We are looking to bring them in and show them on a graphical user interface tutorial.

To see how to load a set of values into a combobox from an SQLite database table, have a look at how to load values into a combobox

How to data cleanse a database table

In Data Analytics, that is a very relevant question, and something I look to implement in most projects, sometimes it is too easy to click the shortcut icon to your spreadsheet application!

Here we are looking to bring a little automation into these videos. Building on How to import data from files and Removing characters from an imported CSV file this video connects to a Microsoft Azure cloud database table, brings in the data with errors on it, fixes the errors and displays the correct output on the screen.

What can this do for organisations?

There are several benefits to automating this step:

  • Less manual intervention if there is a need to fix data issues.
  • Better productivity.
  • Better data flows with no errors and quicker reporting.

 

Moving away from files

The process of moving away from files and into automation has several steps:

  • Be clear on your data needs.
  • Understand what you are trying to achieve.
  • Build a process that is repeatable but can be updated easily.
  • Ensure that you build in data quality checks, helps deliver the better output to the users.

Thanks for stopping by!

Data Analytics Ireland

 

How to save data frame changes to a file

Estimated reading time: 2 minutes

Changing a file is the natural step; tracking those changes are just as important.
Change is part and parcel of life, but in the technology world with the complexity  and interdependency of systems, not effectively been able to track what goes on leads to:

  • Countless hours are trying to figure out where it went wrong.
  • You do not understand what needs fixing.
  • Systems/processes that ultimately work seamlessly, slow down unnecessarily.

In data analysis, as the volumes can be quite large, the human cannot feasibly review a set of data and find out where the underlying problem is. Well, they can, but it would take so long, nothing else would get done. This article by Forbes – predictions-about-data-in-2020-and-the-coming-decade predicts the consumption of data will just be getting bigger.

 

Let the script work, see the log of changes in the output.

How do you remove characters from an imported CSV file, looked at some data cleansing techniques, but there was no way of knowing what was changed other than a visual inspection. Here we introduce the data set into a data frame, change some of the values and show the output on the screen. But more importantly, as we progress through these steps, we are saving the changes as we go along.

The reality is that in large corporate settings, visual inspections would take up too much time and resources. An IT solution to help with giving the vital information required will reduce the data errors happening and allow for a more unobstructed view of how the companies data has changed over time.

 

Where does the trail lead to next?

  • Changes made to data needs a clear way of being able to be tracked.
  • How you captured those changes on your systems, needs to be addressed.
  • Implementing better systems will help you have confidence in your data changes.

Showing audit trail of changes