ValueError: Columns must be same length as key

Estimated reading time: 3 minutes

Are you looking to learn python , and in the process coming across this error and trying to understand why it occurs?

In essence, this usually occurs when you have more than one data frames and in the process of writing your program you are trying to use the data frames and their data, but there is a mismatch in the no of items in each that the program cannot process until it is fixed.

A common scenario where this may happen is when you are joining data frames or splitting out data, these will be demonstrated below.

Scenario 1 – Joining data frames

Where we have df1[[‘a’]] = df2 we are assigning the values on the left side of the equals sign to what is on the right.

When we look at the right-hand side it has three columns, the left-hand side has one.

As a result the error “ValueError: Columns must be same length as key” will appear, as per the below.

import pandas as pd

list1 = [1,2,3]
list2 = [[4,5,6],[7,8,9]]

df1 = pd.DataFrame(list1,columns=['column1'])
df2 = pd.DataFrame(list2,columns=['column2','column3','column4'])

df1[['a']] = df2

The above code throws the below error:

The objective here is to have all the columns from the right-hand side, beside the columns from the left-hand side as follows:

What we have done is make both sides equal regards the no of columns to be shown from df2
Essentially we are taking the column from DF1, and then bringing in the three columns from DF2.
The columna, columnb, columnc below correspond to the three columns in DF2, and will store the data from them.

The fix for this issue is : df1[[‘columna’,’columnb’,’columnc’]] = df2

print (df1)

Scenario 2 – Splitting out data

There may be an occasion where you have a python list, and you need to split out the values of that list into separate columns.

new_list1 = ['1 2 3']
df1_newlist = pd.DataFrame(new_list1,columns=['column1'])

In the above, we have created a list, with three values that are part of one string. Here what we are looking to do is create a new column with the below code:

df1_newlist[["column1"]] = df1_newlist["column1"].str.split(" ", expand=True) #Splitting based on the space between the values.

print(df1_newlist)

When we run the above it throws the following valueerror:

The reason it throws the error is that the logic has three values to be split out into three columns, but we have only defined one column in df1_newlist[[“column1”]]

To fix this, we run the below code:

df1_newlist[["column1","column2","column3"]] = df1_newlist["column1"].str.split(" ", expand=True) #Splitting based on the space between the values.

print(df1_newlist)

This returns the following output, with the problem fixed!

How To Check For Unwanted Characters Using Loops With Python

Estimated reading time: 3 minutes

On this website, we have posted about how to remove unwanted characters from your data and How to remove characters from an imported CSV file both will show you different approaches.

In this blog posting, we are going to approach the process by using loops within a function. Essentially we are going to pass a list and then we are going to loop through the strings to check the data against it.

Step 1 – Import the data and create a data frame

The first bit of work we need to complete is to load the data. Here we create a dictionary with their respective key-value pairs.

In order to prepare the data to be processed through the function in step 2, we then load it into a data frame.

import pandas as pd
#Create a dataset locally
data = {'Number':  ["00&00$000", '111$11111','2222€2222','3333333*33','444444444£','555/55555','666666@666666'],
        'Error' : ['0','0','0','0','0','0','0']}

#Create a dataframe and tell it where to get its values from
df = pd.DataFrame (data, columns = ['Number','Error'])

Step 2 – Create the function that checks for invalid data

This is the main piece of logic that gives the output. As you can see there is a list “L” that is fed to the function run.

One thing to note is that *l is passed to the function, as there is more than one value in the list, otherwise the program would not execute properly.

To start off we create a data frame, which extracts using a regular expression the characters we don’t want to have.

Next, we then need to drop a column that is generated with NAN values, as these are not required.

Then we updated the original data fame with the values that we found.

Just in case if there are any NAN values in this updated column “Error”, we remove them on the next line.

The main next is the loop that creates a new column called “Fix”. This holds the values that will be populated into the fix after the data we don’t want is removed and is data cleansed.

The data we do not need is in str.replace.

#Function to loop through the dataset and see if any of the list values below are found and replace them
def run(*l):
    #This line extracts all the special characters into a new column
    #Using regular expressions it finds values that should not appear
    df2 = df['Number'].str.extract('(\D+|^(0)$)') # New dataframe to get extraction we need
    print(df2)
    df2 = df2.drop(1, axis=1) # Drops the column with NAN in it, not required

    df['Error'] = df2[0] # Updates original dataframe with values that need to be removed.
    #This line removes anything with a null value
    df.dropna(subset=['Error'], inplace=True)
    #This line reads in the list and assigns it a value i, to each element of the list.
    #Each i value assigned also assigns an index number to the list value.
    #The index value is then used to check whether the value associated with it is in the df['Number'] column 
    #and then replaces if found
    for i in l:
        df['Fix']= df['Number'].str.replace(i[0],"").str.replace(i[1],"").str.replace(i[2],"").str.replace(i[3],"") \
        .str.replace(i[4],"").str.replace(i[5],"").str.replace(i[6],"")
        print("Error list to check against")
        print(i[0])
        print(i[1])
        print(i[2])
        print(i[3])
        print(i[4])
        print(i[5])
        print(i[6])
    print(df)

#This is the list of errors you want to check for
l = ['$','*','€','&','£','/','@']

Step 3 – Run the program

To run this program, we just execute the below code. All this does is read in the list “L” to the function “run” and then the output in step 4 is produced

run(l)

Step 4 – Output

Error list to check against
$
*
€
&
£
/
@
          Number Error           Fix
0      00&00$000     &       0000000
1      111$11111     $      11111111
2      2222€2222     €      22222222
3     3333333*33     *     333333333
4     444444444£     £     444444444
5      555/55555     /      55555555
6  666666@666666     @  666666666666

Process finished with exit code 0

TypeError: Array() Argument 1 Must Be A Unicode Character, Not List

In a recent post, we discussed arrays and what they mean and how they differ from lists. When you have a list in an array, one of the things that you need to define is what data type it is. If you don’t then the array will throw the error that you are on this page to resolve.

On the Python.org website, below are the list of values that can be populated to indicate what the data type of a list is in an array.

If you don’t add in the value you will get the below error:

ValueError: bad typecode (must be b, B, u, h, H, i, I, l, L, q, Q, f or d)

Let’s look at an example

import array as test_array

a = test_array.array([1,2,3])

print(a)

Gives an error of:
TypeError: array() argument 1 must be a unicode character, not list

As can be seen, the above logic tries to print a list within an array. The problem here is that an array can only be of one data type, and it has to be specified on the array creation.

How do we fix this?

It is quite simple! Before the list, we simply specify which type code we would like to apply from the above list. In this instance we are going to assign it as “signed int” that being the value “i” as follows:

import array as test_array

a = test_array.array("i",[1,2,3])

print(a)

Prints the following with no error:
array('i', [1, 2, 3])

Process finished with exit code 0

I can change the value “i” to any of the values I want from the above list, just picked that one to show as an example.

How To Run Python Validation From Javascript

Estimated reading time: 6 minutes

More and more, website developers are looking to blend different programming technologies to allow them to use functionality, that maybe is not available to them in the current programming language they utilise.

In previous posts how to pass python variables to Javascript and how to run python directly from javascript, we touched on how to use Python and Javascript interchangeably to pass data.

Here in this blog post, we are going to look at data validation using Python, with data captured by Javascript. The purpose here is to allow another way to validate data other than relying on javascript.

Python Validation run from Javascript

There are a number of reasons for this:

  1. You may want to use Python to process the data and provide output.
  2. The data could be passed from Python into a database, you need to run some checks before you proceed.
  3. Python is your core language, you only use Javascript to capture the data and pass it from a website.

There are four validation checks

  1. Validation if a username exists.
  2. Validation if a number has been entered on a username field.
  3. Validation of a password entered.
  4. A validation check to make sure that an email address entered is in the correct format.

We have two files with the requisite code in them as follows:

  1. app.py ===> As this application is built in Python Flask, this file holds the logic to open the website files, and it holds the Python validation checks.
  2. index.html ====> This file holds the HTML that creates the pages, labels, and input boxes that appear in the web browser. It also includes the Javascript that will capture the data to be passed to Python for validation.

Code Overview – App.PY

Below, I will look to explain what is going on as follows:

The variable Regex is used in the email validation to check if the email entered is correct.

def inputcheck ===> This is just checking the username passed over and performing some validations on it.

def inputvalidation ====> This also is checking the username but looking at if numbers only are entered, or if it is empty.

def passvalidation ====> In this piece of logic, it checks if the password is empty, less than five characters, is numbers only.

def emailvalidation ====> All this is doing is checking if the data received from the Javascript is in the correct email format. It references the regex variable above, which is used to confirm if the format is correct or otherwise.

Within each of the functions, there are if statements which are the core validation checks used by Python.

Also, all the popup menus use ctypes, which allows you to access the windows libraries that hold message boxes, customise them and call within your program.

import json
import ctypes
import re
from flask import request,redirect,url_for

from flask import Flask, render_template

app = Flask(__name__)

regex = '^[a-z0-9]+[\._]?[a-z0-9]+[@]\w+[.]\w{2,3}$'

@app.route('/')
def index():
    return render_template('index.html')


@app.route('/inputcheck', methods=['GET','POST'])
def inputcheck():
    output = request.get_json()
    result = json.loads(output) #this converts the json output to a python dictionary
    username_list = ['joe','john']
    if result['username'] in username_list:
        MessageBox = ctypes.windll.user32.MessageBoxW(0, 'Username ' + result['username'] + ' ' + 'exists', "Username check", 0x00001000)
        return render_template('various.html')
    elif result['username'] == '':
        MessageBox = ctypes.windll.user32.MessageBoxW(None, 'You cannot enter an empty value', 'Username check', 0x00001000)
    else:
        MessageBox = ctypes.windll.user32.MessageBoxW(None, 'Username ' + result['username'] + ' ' + 'does not exist', 'Username check', 0x00001000)

@app.route('/inputvalidation', methods=['GET','POST'])
def inputvalidation():
    output = request.get_json()
    result = json.loads(output) #this converts the json output to a python dictionary
    if result['username'].isdecimal():
        MessageBox = ctypes.windll.user32.MessageBoxW(None, 'Your username cannot be numbers',"Number check", 0x00001000)
    elif result['username'] == '':
        MessageBox = ctypes.windll.user32.MessageBoxW(None, 'The username cannot be empty', "Number check",0x00001000)
    else:
        MessageBox = ctypes.windll.user32.MessageBoxW(None, 'Your username looks ok', "Number check", 0x00001000)
    return render_template('index.html')

@app.route('/passvalidation', methods=['GET','POST'])
def passvalidation():
    output = request.get_json()
    result = json.loads(output) #this converts the json output to a python dictionary
    if result['password'].isdecimal():
        MessageBox = ctypes.windll.user32.MessageBoxW(None, 'Your password cannot be numbers',"Password check", 0x00001000)
    elif result['password'] == '':
        MessageBox = ctypes.windll.user32.MessageBoxW(None, 'The password cannot be empty', "Password empty check",0x00001000)
    elif len(result['password']) < 5:
        MessageBox = ctypes.windll.user32.MessageBoxW(None, 'Your username should be greater than five characters', "Password length check", 0x00001000)
    else:
        MessageBox = ctypes.windll.user32.MessageBoxW(None, 'Your password looks ok', "Number check", 0x00001000)
    return render_template('index.html')

@app.route('/emailvalidation', methods=['GET','POST'])
def emailvalidation():
    output = request.get_json()
    result = json.loads(output) #this converts the json output to a python dictionary
    if re.search(regex, result['email']):
        MessageBox = ctypes.windll.user32.MessageBoxW(None, 'Your email is in the correct format', "Email check", 0x00001000)
    else:
        MessageBox = ctypes.windll.user32.MessageBoxW(None, 'Your email is invalid, please correct', "Email check", 0x00001000)
    return render_template('index.html')



if __name__ == "__main__":
    app.run(debug=True)


Code Overview – Index.html

The below code is the page code, that allows the capturing of data through a web browser HTML page.

Between the <style></style> tags these are the CSS properties that format the labels and buttons on the page.

Within the <div></div> tags are where we create the labels and buttons to show on the screen. Also within these, are the references to the on click event function that should run once the buttons are clicked.

Within the <script></script> tags, this is where the javascript is written which captures the data entered into the input boxes within the <div> tags.

Also, you will see within this javascript, there are sections called $.ajax, which is where the data captured by the javascript is stored, and then passed onto the Python script file (app.py)

Note that in each ajax , url:”/APAGENAME” is referenced. These are the sections within the app.py that the data is passed to, and then the Python logic kicks in and validates the data.

<html lang="en">

<head>

    <title>Data Analytics Ireland</title></head>
<style>
.button1 {
    position: absolute;
    top: 50%;
    left: 55%;
    width: 900px;
    height: 300px;
    margin-left: -300px;
    margin-top: -80px;
}



.labels {
    position: absolute;
    top: 50%;
    left: 55%;
    width: 900px;
    height: 300px;
    margin-left: -300px;
    margin-top: -150px;
}

</style>

<body>



<div class="labels" id="mem1" >
<label  for="username">username:</label> <input type="text" id="username" name="username">
<label for="password">password:</label><input type="text" id="password" name="password">
    <label for="email">email:</label><input type="text" id="email" name="password">
</div>


<div class="button1" id="mem2" >
<button type="submit" onclick='myfunction();'>Click here to validate your username</button>
<button type="submit" onclick='myfunctionval();'>Click here to check if a number</button>
    <button type="submit" onclick='myfunctionpass();'>Click here to check password</button>
      <button type="submit" onclick='myfunctionemail();'>Click here to check your email</button>
</div>


<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.5/jquery.min.js"></script>

<script>
    function myfunction() {

        const username = document.getElementById("username").value;
        const password = document.getElementById("password").value;


        const dict_values = {username, password} //Pass the javascript variables to a dictionary.
        const s = JSON.stringify(dict_values); // Stringify converts a JavaScript object or value to JSON.
        console.log(s); // Prints the variables to console window, which are in the JSON format
        //window.alert(s)
        $.ajax({
            url:"/inputcheck",
            type:"POST",
            contentType: "application/json",
            data: JSON.stringify(s)});
}
</script>
<script>
    function myfunctionval() {

        const username = document.getElementById("username").value;
        const password = document.getElementById("password").value;


        const dict_values = {username, password} //Pass the javascript variables to a dictionary.
        const s = JSON.stringify(dict_values); // Stringify converts a JavaScript object or value to JSON.
        console.log(s); // Prints the variables to console window, which are in the JSON format
        //window.alert(s)
        $.ajax({
            url:"/inputvalidation",
            type:"POST",
            contentType: "application/json",
            data: JSON.stringify(s)});
}
</script>

<script>
    function myfunctionpass() {

        const username = document.getElementById("username").value;
        const password = document.getElementById("password").value;


        const dict_values = {username, password} //Pass the javascript variables to a dictionary.
        const s = JSON.stringify(dict_values); // Stringify converts a JavaScript object or value to JSON.
        console.log(s); // Prints the variables to console window, which are in the JSON format
        //window.alert(s)
        $.ajax({
            url:"/passvalidation",
            type:"POST",
            contentType: "application/json",
            data: JSON.stringify(s)});
}
</script>

<script>
    function myfunctionemail() {

        const email = document.getElementById("email").value;


        const dict_values = {email} //Pass the javascript variables to a dictionary.
        const s = JSON.stringify(dict_values); // Stringify converts a JavaScript object or value to JSON.
        console.log(s); // Prints the variables to console window, which are in the JSON format
        //window.alert(s)
        $.ajax({
            url:"/emailvalidation",
            type:"POST",
            contentType: "application/json",
            data: JSON.stringify(s)});
}
</script>




</body>
</html>

How to Pass Python Variables to Javascript

Estimated reading time: 4 minutes

In our recent blog posting How to Pass a Javascript Variable to Python using JSON, we demonstrated how to easily use AJAX to pass whatever data you wanted and then manipulate it with Python.

In this blog posting, we are going to show how to do this the other way around. The scenario is that you have an application and or website that wants to use data generated through Python, but let Javascript then use it within the application.

As Python can be connected to numerous databases and files ( txt, excel) etc, this piece of logic is very useful for the programmer looking to integrate both programming languages.

Let’s start looking at the code, and see how this can be achieved.

Step 1 – What Files are generated?

This program uses Python Flask to create a web page, that has a drop-down menu. The two files used to generate this are as follows:

(A) app.py – This is the python file that creates a website and loads a template HTML file as outlined below.

(B) Index.html – This is the template file that loads into the browser and runs all the javascript. The javascript loaded here also loads the python data passed over from app.py

Step 2 – APP.PY code overview

The Python library that enables webpage creation is called Flask, and as can be seen below it has to be imported.

In addition, we need to also import render_template which tells the program to go to the templates folder and load “Index.HTML”

The variable that is been passed to JavaScript is called name, and these are the values that you will see in the web browser when it is loaded.

from flask import Flask, render_template

app = Flask(__name__)

@app.route('/')
def index():
    name = ['Joe','John','Jim','Paul','Niall','Tom']
    return render_template('index.html', name=name)

if __name__ == "__main__":
    app.run(debug=True)

Step 3 – Index.HTML overview

Here is the template HTML file that runs in the browser. You can add CSS etc to this to make it look nicer and more user friendly.

As you can see it has the usual HTML tags appear as part of a website.

Well look at some of the code further:

In this bit <select id =’select’> </select>, this is the dropdown menu that will appear when Index.html is opened. It will store all the values passed from python. Note that its id is “select”, this will be used later on.

The main parts to focus on next is between <script></script>. This is what reads in the python data and populates it to the dropdown menu.

In step 2 we mentioned that there was a variable called “name”, with values to be passed over.

This is acheived on this line:

var select = document.getElementById(“select”), test = {{ name | tojson }};

Notice that name appears here, and this is referencing back to the exact same value that was discussed in step 2.

For the rest of the lines, I have explained with comments what each does.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Pass Python variable to Javascript</title>
</head>
<body>

<select id ='selectvalue'>
</select>
<script>
    //name = ['Joe','John','Jim','Paul','Niall','Tom']

    var selectvalue = document.getElementById("selectvalue"), test = {{ name | tojson }};
    //The increment operator (++) increments (adds one to) its operand and returns a value.
    for(var i = 0; i < test.length; i++) // This line checks for the length of the data you feeding in i.e the no of items
         {
var selection = document.createElement("OPTION"), // This line creates a variable to store the different values fed in from the JSON object "TEST"
txt = document.createTextNode(test[i]); // This just reads each value from the test JSON variable above
selection.appendChild(txt); // This line appends each value as it is read.
selection.setAttribute("value",test[i]); // This line sets each value read in as a value for the drop down
selectvalue.insertBefore(selection,selectvalue.lastChild); //This reads eah value into the dropdown based on the order in the "TEST" above.
 }
</script>
</body>
</html>

Step 4 – What the output looks like!

From step 2, these are values we asked to be used in Javascript to populate a dropdown:

name = [‘Joe’,’John’,’Jim’,’Paul’,’Niall’,’Tom’]

Python variable passed to Javascript

How to pass multiple lists to a function and compare

Estimated reading time: 3 minutes

Have you ever been faced with a situation where you have multiple lists you want to compare, and need a function that will read them in and show you what is different between them all?

In this scenario, this can be handy where you have large data sets and need a quick way to find those differences and fix them where appropriate.

Comparing two lists

Let’s look and see what is going on below.

First of all, we have defined two lists. The only difference between the two is that one has a value of six, the other does not.

Next, we have the function “comparelists”. What this is doing is taking in the two lists as parameters (a,b), and then processing them.

The lists are passed as arguments to the parameters in this line ===> comparelists(list1,list2)

The parameters a are assigned to list1, and b is assigned to list2.

The main bit of the function is the list comprehension, and it is doing the following:

  1. x for x is basically creating a variable x, and starting a loop that goes through all the values of b.
  2. Each iteration of x is stored and compared with a.
  3. “If x not in a” completes the comparison and if true returns the value, otherwise moves to the next value.

As a result of this logic, it can be seen that six is the only value returned, and this is what we are expecting based on a visual inspection.

#Reading in two lists
list1 = [1,2,3,4,5] # parameter a below
list2 = [1,2,3,4,5,6] # parameter b below

def comparelists(a,b):
    z = [x for x in b if x not in a] #list comprehension
    print(z)

comparelists(list1,list2)

Output:

[6]

Comparing more than two lists

In the logic above, we saw that two lists can be compared. But what if you want to compare one list against two other lists?

That is easily done, with some tweaking to the code.

As before the three lists are defined, created as arguments in comparelists(list1,list2,list3), and then passed to the function parameters a,b,c.

The only difference in this logic is that the list comprehension is written a little different as follows:

  1. x for x is basically creating a variable x, and starting a loop that goes through all the values of b. ====> This is the same as above
  2. Each iteration of x is stored and compared with a and c ===> This line is different as comparing to two lists now.
  3. “if x not in a and x not in c” ====> Here we have two comparisons
    • Checking for the value x from b in a
    • Checking for the value x from b in c
  4. The value of six is correct as it is not in either a or c.
  5. Note the if statement is specific as to what to check for with the “and” statement. This can be changed to suit your needs.
list1 = [1,2,3,4,5] # parameter a below
list2 = [1,2,3,4,5,6] # parameter b below
list3 = [1,2,3,7,8,9] # parameter c below

def comparelists(a,b,c):
    z = [x for x in b
         if x not in a and x not in c] #list comprehension
    print(z)

comparelists(list1,list2,list3)

Output:
[6]

How to use parameters in Python

Estimated reading time: 5 minutes

Are you using a function and looking to pass values into it to be processed? Well if this is the case you are trying to pass arguments to that function’s parameters so they will return a value for you.

Parameters are just values that are associated with the function. For each function, you are going to have one or more parameters it is expecting to process.

As a result, there are a number of ways they are created, let’s explore how this can be achieved.

Passing arguments directly into a function parameters

In a lot of instances, the argument values to be processed are going to be passed into the function parameters, which then executes some logic and returns a value.

Typical ways this may happen:

  1. A password is entered ( the inputted argument), the function then confirms if it is valid or not.
  2. You may need to check if a value entered ( the inputted argument) is above or below a certain value. It is passed to a function parameter and then the function does the necessary checks, and confirms the result in the output.

The above are two examples, but let’s see this in action.

Below we have a function. The purpose is to take in two arguments passed to it and conduct a calculation on them.

The calculation happens in the variable number 1 or number 2.

Variable number 1 and variable number 2 get their values from parameters a and b respectively.

The line multiplenumber(1,2) is just calling the function and passing the arguments to the parameters a,b which then processes them and the output is printed.

def multiplenumber(a,b):
    number1 = a*2
    number2 = b*3
    print(number1,number2) #===> These values are different to the variables below as they happen as a result of the processing inside the function.

number1=1
number2=2
multiplenumber(number1,number2)
print(number1,number2) #===> These values are outside the function and do not have any relationship to the variables inside the function above, hence are different.

Result:
2 6
1 2

The result of the above piece of logic is called “passing by value“, which essentially means the data is passed from the variables ( number 1 and Number 2) to the function parameters, but when the function is completed processing, it does not impact the original variables outside the function.

This is because the original variables are immutable.

Passing arguments directly into function parameters from a list

In the above example, as outlined, there will be no impact on the variables that pass the arguments into the function, they remain the same.

On the other hand, if the original source of the arguments is a mutable object, it will have its values changed.

Let’s look at a situation where this may occur.

def addnumber(a):
    number=['3']
    a.append(number)
    print(a) #===> This prints the original value of a with the addition of the value 3

number1=[1,2]
addnumber(number1)
print(number1) #===> As number1 is passed to the function, and number1 is allowed to be changed then this will print out the new list value once the function has completed processing.

Result:
[1, 2, ['3']]
[1, 2, ['3']]

So in this instance, as number1 is a list, and this is mutable, its value will change once the function completes its processing.

This can also be called “passing by reference“.

Passing arguments between functions

In another video, we completed called how to pass data between functions this discusses how you can pass the data between two functions and print it out.

Effectively in the below the arguments are name, country, city, and these values are passed to functionb

def functiona():
    name = "Joseph"
    country = "Mars"
    city = "atlantis"
    #this captures the three variables above
    return name,country,city
functiona()

def functionb():
    myname,mycountry,mycity = functiona()
    print(myname)
    print(mycountry)
    print(mycity)

functionb()

Result:
Joseph
Mars
atlantis

Passing arguments using *args as the parameter

In order to understand this, before we look at some code we need to understand what are *args?

*args essentially allows you to pass multiple positional arguments to a function, and this can vary over time.

As a result, you don’t have to assign the positional arguments to more than one variable. The *args take in all the arguments, no matter how long or short they are and then return a value.

def addnumbers(*args):
    result = 2
    for x in args:
        result = result * x
    return result

print(addnumbers(1,2,3))

Result:
12

Passing arguments using **kwargs as the parameter

In contrast to the above, this will process a set of values that have a name attached to them.

As can be seen in the previous example the arguments passed had no name attached to them.

Let’s look closely at an example. In this scenario, the arguments are assigned a value of a,b,c.

def addnumbers(**kwargs):
    result = 2
    for x in kwargs.values():
        result = result * x
    return result

print(addnumbers(a=1,b=2,c=3))

Result:
12

Here it is clear that the logic takes in each value of a,b,c processes it and then returns the result required.

In essence, both *args and **kwargs give you the same answer, it just depends on other parts of your code and how it is processing your data.

For example, in the **kwargs it could be that the values of a,b,c could be populated by other variables within the program.

So in that scenario, the variable that has been passed on can change and not be static.

So in summary:

a. Arguments can be passed multiple ways to functions, the parameters then just process them.

b. Parameters also don’t have to be assigned to a variable, they can be read in through *args.

c. On the other hand you can have multiple arguments read in at once by using the **kwargs.

TypeError: List Indices Must Be Integers Or Slices, Not Tuple

Estimated reading time: 3 minutes

When working with Python lists in your data analytics projects, when you trying to read the data, a common problem occurs if you have a list of lists, and it is not properly formatted.

In this instance, Python will not be able to read one or more lists and as a result, will throw this error.

In order to understand how this problem occurs, we need to understand how to create a list of lists.

How to create a lists of lists

Let’s look at a simple list:

a = [1,2,3]
print(a)
print(type(a))

Result:
[1, 2, 3]
<class 'list'>

Let’s create a second list called b:

b = [4,5,6]
print(b)
print(type(b))

Result:
[4, 5, 6]
<class 'list'>

So if we want to join the lists together into one list ( hence a list of lists) then:

a = [1,2,3]
b = [4,5,6]

list_of_lists = []
list_of_lists.append(a)
list_of_lists.append(b)
print(list_of_lists)
print(type(list_of_lists))

Result:
[[1, 2, 3], [4, 5, 6]]
<class 'list'>

So as can be seen the two lists are contained within a master list called “list_of_lists”.

So why does the error list indices must be integers or slices, not tuple occur?

Reason 1 ( Missing commas between lists)

If you manually type them in and forget the comma between the lists this will cause your problem:

a=[[1,2,3][4,5,6]]
print(a)

Result (Error):
Traceback (most recent call last):
  line 10, in <module>
    a=[[1,2,3][4,5,6]]
TypeError: list indices must be integers or slices, not tuple

But if you put a comma between the two lists then it returns no error:

a=[[1,2,3],[4,5,6]]
print(a)

Result (no error):
[[1, 2, 3], [4, 5, 6]]
Process finished with exit code 0

Reason 2 ( Comma in the wrong place)

Sometimes you have a list, and you only want to bring back some elements of the list, but not others:

In the below, we are trying to bring back the first two elements of the list.

a= [1,2,3,4,5]
print(a[0:,2])

Result:
Traceback (most recent call last):
   line 14, in <module>
    print(a[0:,2])
TypeError: list indices must be integers or slices, not tuple

The reason that the same error happens is the additional comma in a[0:,2], causes the error to appear as Python does not know how to process it.

This is easily fixed by removing the additional comma as follows:

a= [1,2,3,4,5]
print(a[0:2])

Result:
[1, 2]
Process finished with exit code 0

So why is there a mention of tuples in the error output?

The final piece of the jigsaw needs to understand why there is a reference to a tuple in the error output?

If we return to a looking at a list of lists and look at their index values:

a=[[1,2,3],[4,5,6]]
z = [index for index, value in enumerate(a)]
print(z)

Result:
[0, 1]
Process finished with exit code 0

As can be seen, the index values are 0,1, which is correct.

As before removing the comma gives the error we are trying to solve for:

a=[[1,2,3][4,5,6]]
z = [index for index, value in enumerate(a)]
print(z)

Result:
Traceback (most recent call last):
  line 16, in <module>
    a=[[1,2,3][4,5,6]]
TypeError: list indices must be integers or slices, not tuple

BUT the reference to the tuple comes from the fact that when you pass two arguments (say an index value) a Tuple is created, but in this instance, as the comma is missing the tuple is not created and the error is called.

This stems from the __getitem__ for a list built-in class cannot deal with tuple arguments that are not integers ( i.e. 0,1 like we have returned here) , so the error is thrown, it is looking for the two index values to pass into a tuple.

How to Compare Column Headers in CSV to a List in Python

Estimated reading time: 3 minutes

So you have numerous different automation projects in Python. In order to ensure a clean and smooth straight-through processing, checks need to be made to ensure what was received is in the right format.

Most but not all files used in an automated process will be in the CSV format. It is important there that the column headers in these files are correct so you can process the file correctly.

This ensures a rigorous process that has errors limited.

How to compare the headers

The first step would be to load the data into a Pandas data frame:

import pandas as pd

df = pd.read_csv("csv_import.csv") #===> Include the headers
print(df)

The actual original file is as follows:

Next we need to make sure that we have a list that we can compare to:

header_list = ['Name','Address_1','Address_2','Address_3','Address_4','City','Country']

The next step will allow us to save the headers imported in the file to a variable:

import_headers = df.axes[1] #==> 1 is to identify columns
print(import_headers)

Note that the axis chosen was 1, and this is what Python recognises as the column axes.

Finally we will apply a loop as follows:

a = [i for i in import_headers if i not in header_list]
print(a)

In this loop, the variable “a” is taking the value “i” which represents each value in the import_headers variable and through a loop checks each one against the header_list to see if it is in it.

It then prints out the values not found.

Pulling this all together gives:

import pandas as pd

df = pd.read_csv("csv_import.csv") #===> Include the headers
print(df)

#Expected values to receive in CSV file
header_list = ['Name','Address_1','Address_2','Address_3','Address_4','City','Country']

import_headers = df.axes[1] #==> 1 is to identify columns
print(import_headers)


a = [i for i in import_headers if i not in header_list]
print(a)

Resulting in the following output:

As can be seen the addresses below where found not to be valid, as they where not contained within our check list “header_list”

TypeError: ‘list’ object is not an iterator

We have covered off many TypeErrors on this website, here we will go through which using a list with and it is not an iterator gives you errors.

In order to understand this error better, we need to first understand what is an iterator in Python?

An iterator is a Python object that has the following characteristics:

  • You can count the no of values that are contained within it.
  • It also can be iterated through, so you need to apply an iteration method to it.

How does this error occur?

Normally this error occurs when you try to iterate over a list, but you have not made the list iterable.

There are two things required to make this happen:

(A) The iter() returns an iterator object

(B) The next() method moves to the next value.

Without both the code will fail and the error you are about will occur!

In the below code we have a list:

a = ['q', 'w', 'e', 'r', 't', 'y']

with the following:

b = next(a)
b = next(a)
b = next(a)
b = next(a)
b = next(a)
b = next(a)

As can be seen in the above code we have one component for the iteration , we expect two as per the above.

As a result we get the error:

Traceback (most recent call last):
  File "list_object_is_not_an_iterator.py", line 13, in <module>
    b = next(a)
TypeError: 'list' object is not an iterator

In order to fix this ,all we need to do is apply the iterator to the list as follows:

a = iter(['q', 'w', 'e', 'r', 't', 'y']) ====> We added in the iter() here, enclosing the list within it

b = next(a)
b = next(a)
b = next(a)
b = next(a)
b = next(a)
b = next(a)
#b = next(a)


print(b)

Giving output:
y

As a result of this, we now have the two required methods that will not give this error.

What is going on within the iterator?

In the above code we have asked to print b. What the iterator is doing is going to the first value of b, in this case q and print.

But because we have a variable b on multiple lines, with the method “next()” in it, the logic is moving through each value of the list till it gets to the end.

What can be done though is , reduce the length of the returned b variables to print as follows:

a = iter(['q', 'w', 'e', 'r', 't', 'y'])
b = next(a)
print(b)
returns:
q

BUT
a = iter(['q', 'w', 'e', 'r', 't', 'y'])
b = next(a)
b = next(a)
print(b)
returns:
w

As can be seen it returns the next value in the list. You can keep adding the b variables.

What happens when you get to the end of the list?

So now we have the below, and we are returning the last value:

a = iter(['q', 'w', 'e', 'r', 't', 'y'])
b = next(a)
b = next(a)
b = next(a)
b = next(a)
b = next(a)
b = next(a)

Returns:
y

The reason for this is that we have the required no of variables with the next method, which equals the length of the list.

If we add in one more b variable:

a = iter(['q', 'w', 'e', 'r', 't', 'y'])
b = next(a)
b = next(a)
b = next(a)
b = next(a)
b = next(a)
b = next(a)
b = next(a) ===> Additional b variable

Returns: 
Traceback (most recent call last):
  File "list_object_is_not_an_iterator.py", line 19, in <module>
    b = next(a)
StopIteration

The purpose of StopIteration is to not allow a continuous loop and recognise that the end of the list has been reached.

Implementing Iterators

Iterators could be used in the following circumstances:

(A) You have a defined list of object values to work with.

(B) If sequence is important an iterator will help to process values in the order they appear in a list.