In a recent post, we discussed arrays and what they mean and how they differ from lists. When you have a list in an array, one of the things that you need to define is what data type it is. If you don’t then the array will throw the error that you are on this page to resolve.
On the Python.org website, below are the list of values that can be populated to indicate what the data type of a list is in an array.
If you don’t add in the value you will get the below error:
ValueError: bad typecode (must be b, B, u, h, H, i, I, l, L, q, Q, f or d)
Let’s look at an example
import array as test_array
a = test_array.array([1,2,3])
print(a)
Gives an error of:
TypeError: array() argument 1 must be a unicode character, not list
As can be seen, the above logic tries to print a list within an array. The problem here is that an array can only be of one data type, and it has to be specified on the array creation.
How do we fix this?
It is quite simple! Before the list, we simply specify which type code we would like to apply from the above list. In this instance we are going to assign it as “signed int” that being the value “i” as follows:
import array as test_array
a = test_array.array("i",[1,2,3])
print(a)
Prints the following with no error:
array('i', [1, 2, 3])
Process finished with exit code 0
I can change the value “i” to any of the values I want from the above list, just picked that one to show as an example.
In this blog posting, we are going to show how to do this the other way around. The scenario is that you have an application and or website that wants to use data generated through Python, but let Javascript then use it within the application.
As Python can be connected to numerous databases and files ( txt, excel) etc, this piece of logic is very useful for the programmer looking to integrate both programming languages.
Let’s start looking at the code, and see how this can be achieved.
Step 1 – What Files are generated?
This program uses Python Flask to create a web page, that has a drop-down menu. The two files used to generate this are as follows:
(A) app.py – This is the python file that creates a website and loads a template HTML file as outlined below.
(B) Index.html – This is the template file that loads into the browser and runs all the javascript. The javascript loaded here also loads the python data passed over from app.py
Step 2 – APP.PY code overview
The Python library that enables webpage creation is called Flask, and as can be seen below it has to be imported.
In addition, we need to also import render_template which tells the program to go to the templates folder and load “Index.HTML”
The variable that is been passed to JavaScript is called name, and these are the values that you will see in the web browser when it is loaded.
from flask import Flask, render_template
app = Flask(__name__)
@app.route('/')
def index():
name = ['Joe','John','Jim','Paul','Niall','Tom']
return render_template('index.html', name=name)
if __name__ == "__main__":
app.run(debug=True)
Step 3 – Index.HTML overview
Here is the template HTML file that runs in the browser. You can add CSS etc to this to make it look nicer and more user friendly.
As you can see it has the usual HTML tags appear as part of a website.
Well look at some of the code further:
In this bit <select id =’select’> </select>, this is the dropdown menu that will appear when Index.html is opened. It will store all the values passed from python. Note that its id is “select”, this will be used later on.
The main parts to focus on next is between <script></script>. This is what reads in the python data and populates it to the dropdown menu.
In step 2 we mentioned that there was a variable called “name”, with values to be passed over.
This is acheived on this line:
var select = document.getElementById(“select”), test = {{ name | tojson }};
Notice that name appears here, and this is referencing back to the exact same value that was discussed in step 2.
For the rest of the lines, I have explained with comments what each does.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Pass Python variable to Javascript</title>
</head>
<body>
<select id ='selectvalue'>
</select>
<script>
//name = ['Joe','John','Jim','Paul','Niall','Tom']
var selectvalue = document.getElementById("selectvalue"), test = {{ name | tojson }};
//The increment operator (++) increments (adds one to) its operand and returns a value.
for(var i = 0; i < test.length; i++) // This line checks for the length of the data you feeding in i.e the no of items
{
var selection = document.createElement("OPTION"), // This line creates a variable to store the different values fed in from the JSON object "TEST"
txt = document.createTextNode(test[i]); // This just reads each value from the test JSON variable above
selection.appendChild(txt); // This line appends each value as it is read.
selection.setAttribute("value",test[i]); // This line sets each value read in as a value for the drop down
selectvalue.insertBefore(selection,selectvalue.lastChild); //This reads eah value into the dropdown based on the order in the "TEST" above.
}
</script>
</body>
</html>
Step 4 – What the output looks like!
From step 2, these are values we asked to be used in Javascript to populate a dropdown:
Have you ever been faced with a situation where you have multiple lists you want to compare, and need a function that will read them in and show you what is different between them all?
In this scenario, this can be handy where you have large data sets and need a quick way to find those differences and fix them where appropriate.
Comparing two lists
Let’s look and see what is going on below.
First of all, we have defined two lists. The only difference between the two is that one has a value of six, the other does not.
Next, we have the function “comparelists”. What this is doing is taking in the two lists as parameters (a,b), and then processing them.
The lists are passed as arguments to the parameters in this line ===> comparelists(list1,list2)
The parameters a are assigned to list1, and b is assigned to list2.
The main bit of the function is the list comprehension, and it is doing the following:
x for x is basically creating a variable x, and starting a loop that goes through all the values of b.
Each iteration of x is stored and compared with a.
“If x not in a” completes the comparison and if true returns the value, otherwise moves to the next value.
As a result of this logic, it can be seen that six is the only value returned, and this is what we are expecting based on a visual inspection.
#Reading in two lists
list1 = [1,2,3,4,5] # parameter a below
list2 = [1,2,3,4,5,6] # parameter b below
def comparelists(a,b):
z = [x for x in b if x not in a] #list comprehension
print(z)
comparelists(list1,list2)
Output:
[6]
Comparing more than two lists
In the logic above, we saw that two lists can be compared. But what if you want to compare one list against two other lists?
That is easily done, with some tweaking to the code.
As before the three lists are defined, created as arguments in comparelists(list1,list2,list3), and then passed to the function parameters a,b,c.
The only difference in this logic is that the list comprehension is written a little different as follows:
x for x is basically creating a variable x, and starting a loop that goes through all the values of b. ====> This is the same as above
Each iteration of x is stored and compared with a and c ===> This line is different as comparing to two lists now.
“if x not in a and x not in c” ====> Here we have two comparisons
Checking for the value x from b in a
Checking for the value x from b in c
The value of six is correct as it is not in either a or c.
Note the if statement is specific as to what to check for with the “and” statement. This can be changed to suit your needs.
list1 = [1,2,3,4,5] # parameter a below
list2 = [1,2,3,4,5,6] # parameter b below
list3 = [1,2,3,7,8,9] # parameter c below
def comparelists(a,b,c):
z = [x for x in b
if x not in a and x not in c] #list comprehension
print(z)
comparelists(list1,list2,list3)
Output:
[6]
# Use numpy
import numpy as np
a = np.array([1,2,3,4])
print(a)
print(type(a))
print(a.dtype)
Output:
[1 2 3 4]
<class 'numpy.ndarray'>
int32
If we have one of the values as a string, all the other values are converted to a string , as follows:
# Use numpy
import numpy as np
a = np.array([1,2,3,'4'])
print(a)
print(type(a))
print(a.dtype)
Output:
['1' '2' '3' '4']
<class 'numpy.ndarray'>
<U11
# <U11 - When this happens, it means one of the values was returning as a string
# All the other values in the array as a result are converted to strings
(B) Use array
import array as test_array
a = test_array.array('i',[1,2,3])
print(a)
print(type(a))
Output:
array('i', [1, 2, 3])
<class 'array.array'>
On the Python.org website, below are the list of values that can be populated into the above program, depending on what your need is:
When should I use arrays?
It really depends on the nature of your python program, but below are some examples that may help you make a decision:
(A) Many variables of the same type : There maybe a scenario where you have to create an array to store data that is of the same data type. For example you have a list of codes to look up against, which are all integers.
(B) Faster and more efficient : If speed is what you are looking for using arrays, will help improve performance of your computer program, using lists is much slower.
(C) Compactability and efficiency : If the nature of your program needs to store large amounts of data that needs to be accessed quickly , then this would be a good reason to use them.
(D) Ability to retrieve data quickly through indexing: As arrays have index values associated with them, they data can be easiy retrieved.
(E) Need to compute some mathematical values: Arrays are excellent for any numerical operations you need to complete, as the level of coding is minimal.
import array as test_array
a = test_array.array('i',[1,2,3])
mydivider = 2
mynewlist = [x / mydivider for x in a]
print(mynewlist)
result:
[0.5, 1.0, 1.5]
So in summary:
Speed , efficiency and ease of use are the main reasons to use an array.
So you have numerous different automation projects in Python. In order to ensure a clean and smooth straight-through processing, checks need to be made to ensure what was received is in the right format.
Most but not all files used in an automated process will be in the CSV format. It is important there that the column headers in these files are correct so you can process the file correctly.
This ensures a rigorous process that has errors limited.
The next step will allow us to save the headers imported in the file to a variable:
import_headers = df.axes[1] #==> 1 is to identify columns print(import_headers)
Note that the axis chosen was 1, and this is what Python recognises as the column axes.
Finally we will apply a loop as follows:
a = [i for i in import_headers if i not in header_list] print(a)
In this loop, the variable “a” is taking the value “i” which represents each value in the import_headers variable and through a loop checks each one against the header_list to see if it is in it.
It then prints out the values not found.
Pulling this all together gives:
import pandas as pd
df = pd.read_csv("csv_import.csv") #===> Include the headers
print(df)
#Expected values to receive in CSV file
header_list = ['Name','Address_1','Address_2','Address_3','Address_4','City','Country']
import_headers = df.axes[1] #==> 1 is to identify columns
print(import_headers)
a = [i for i in import_headers if i not in header_list]
print(a)
Resulting in the following output:
As can be seen the addresses below where found not to be valid, as they where not contained within our check list “header_list”
Are you working with lists and getting the error IndexError: list index out of range while using Python? There is a very simple explanation to this, and its fix is very easy.
First of all lets understand what is going on with the list.
Traceback (most recent call last):
File "C:/Users/haugh/OneDrive/dataanalyticsireland/YOUTUBE/IndexError_list_index_out_of_range/INDEX_ERROR_LIST_INDEX_OUT_OF_RANGE.py", line 4, in <module>
print(data[4])
IndexError: list index out of range
Lists and their index values
In the below list, we have outputted its values and index values.
data = ['a','b','c','d']
for (i,item) in enumerate(data, start=0): #===> Loops through list and applies index values starting at zero
print(i,item)
Output:
0 a
1 b
2 c
3 d
Process finished with exit code 0
As can be seen the program returns the list values and their indexes. Note that the index starts at zero as we have set start=0.
Start=0 can be set to any value you like, as can be seen here:
data = ['a','b','c','d']
for (i,item) in enumerate(data, start=1): #===> Loops through list and applies index values starting at zero
print(i,item)
Output:
1 a
2 b
3 c
4 d
Process finished with exit code 0
OR
data = ['a','b','c','d']
for (i,item) in enumerate(data, start=22): #===> Loops through list and applies index values starting at zero
print(i,item)
Output:
22 a
23 b
24 c
25 d
Process finished with exit code 0
The purpose of the index value is to tell the program where to start its index from, if left empty it starts at zero.
Lists and the no of index values
In the above examples the index values all occur on four rows.
This is important as when you are looping through the rows, it will not go beyond the length of the rows.
So in this example the enumerate function specifically counts the no rows and stores the index values with each, and then loops through the list till it hits the last one, without error.
data = ['a','b','c','d']
for (i,item) in enumerate(data, start=0): #===> Loops through list and applies index values starting at zero
print(i,item)
Output:
0 a
1 b
2 c
3 d
Process finished with exit code 0
How to fix the error IndexError: list index out of range
So the reason we get the below is that the line print(data[4]) is looking for the row with index value 4, but we know that from observation that does not exist.
To fix this we would change the value 4 in print(data[4]) to any of 0,1,2,3, as they are the index values associated with the list.
data = ['a','b','c','d']
for (i,item) in enumerate(data, start=0): #===> Loops through list and applies index values starting at zero
print(i,item)
print(data[4])
Output:
Traceback (most recent call last):
File "C:/Users/haugh/OneDrive/dataanalyticsireland/YOUTUBE/IndexError_list_index_out_of_range/INDEX_ERROR_LIST_INDEX_OUT_OF_RANGE.py", line 4, in <module>
print(data[4])
IndexError: list index out of range
0 a
1 b
2 c
3 d
Applying a correct valid index value:
data = ['a','b','c','d']
for (i,item) in enumerate(data, start=0): #===> Loops through list and applies index values starting at zero
print(i,item)
print(data[3])
Yields with no error:
0 a
1 b
2 c
3 d
d
So in summary when working with lists and their index values it is important:
(A) Understand the length of your list.
(B) Where your index values start and finish.
This error is easily fixable, but in your code you just need to make sure that you referencing values that are in the range of your index values.
Ever had a process where you received a set of data, and it took a bit of effort to cleanse the data, so it looks the way you want it?
The world of data processing and data exchanged between servers and organisations needs careful attention, one person’s idea of clean data might not be other persons, hence the difference between the two can lead to data issues.
Experian had an excellent article What is data cleansing? in that they talk about several factors about data:
It could be incorrect
and incomplete
and Duplicated
One of the things they highlighted also is that under GDPR, organisations have to deal with more focus on data been accurate, complete and up to date.
We are putting together several videos in this area over time, so you will start to see them as they go up.
Please like and share through the social media buttons shared on the page here, thanks for watching!
Following on from our post on how to use Python lists have you ever wondered how to sort lists for your Python project?
Our latest video on lists will go through some of the techniques available so that you can get an idea of how to structure your data and sort.
Getting to understand how to implement
In this latest video we will look at:
sort() method
sorted() function
sorting a list through a function
Adding in those extra bits to help make the process smoother
Have you thought about sorting ascending/descending?
There is also a discussion on this topic as well, and while an index is available for the list, which you may feel does not merit sorting, there could be other logical reasons to implement sorting.
Leaving out the reverse = True/False in the sorted method can have an impact, though if you require it left out of the list you have created, automatic ascending will be the default.
On this channel, we have discussed a number of different ways to manage your data. In thinking about sorting a list, why would you want to do this?
Some common reasons are:
To visually see if there are duplicates, either on the screen or printed out.
If other objects are dependant on the list, say a combo box, then having duplicates visible can help to reduce the size of their contents.
Iteration – If you are looking to iterate over a list, it will be quicker if it is sorted.
If you want to learn about lists, using them, and how how they can be iterated over, why not visit Data Analytics Ireland YouTube channel, there are lots of videos there that will help explain the concepts discussed here further.
To get some more links on this topic click here python sort method, it is a blog posting from our website that has some useful links and explanations for you.
Python lists are used extensively in projects, as a result it is important to understand their structure.
Some of the things they can be used for:
Lookup values for comparisons.
Passing data to them to store to be referenced elsewhere.
As part of a loop, store values that have been found through the loop logic.
With methods are associated with lists?
Append – Add values to the end of the list
Extend – adds values from an iterable object to the end of the list.
Insert – You can insert an item to a certain position in a list.
Remove – Remove the first value in a list that has a value that was asked to be looked for.
Pop – This also removes a value at a certain position and returns, consequently if no position is specified then it removes the last item and returns it.
Clear – Removes all items from the list.
Index – returns the index value of the first item found that was asked to be searched for.
Count – returns the number of times an item that was searched or was found in a list.
Sort – Sorts the items in the list.
Reverse – This reverses the items in the list.
Copy – This makes a copy of the list.
What are the properties of a list?
The data type has the following attributes, that make it really useful for a vast array of scenarios:
They are ordered – Whatever order the list is a unique characteristic of the list, furthermore changing the order makes it a different list.
You can use their index to access the value.
They are mutable, meaning you can apply any of the above methods on them.
They can contain strings, integers etc, accordingly, there is no restriction on what can be in the list.
Check out the below video playlist from our YouTube channel, they will help explain more about lists:
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Cookie settingsACCEPT
Privacy & Cookies Policy
Privacy Overview
This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
