In this second video how to groupby using pandas and as part of expanding the data analytics information of this website, we are looking to explain how you can use a groupby selection but only using the column values and not the column names.
Below we import our data into a dataframe, and then group as follows:
Aggregate function
Using the cut function and assigning values to bins.
Assigning labels to the data frame output based on the bin values.
Why would you want to use Pandas groupby and column values?
This video looks to help understand the why going by values might be easier than column names:
Column names can change from project to project, using by values allows easy implementation of getting the output regardless of the names used.
You could apply this to any Python class, and as long as you can inherit will allow the code to run smoothly.
Implementing by value allows a clear understanding of the desired output as the values are clearly understood to generate what is required.
You need to understand how data within your data set falls within a particular cohort:
This use of values in different programs just needs to change, the underlying logic remains the same.
Using column names still means that to group them, the logic still needs to be written.
In this first video about pandas groupby and as part of expanding the data analytics information of this website, we are looking to explain how you can use a groupby selection to sort your data into similar datasets better so they can be better analysed. In the video below, we import our data into a dataframe, and then group as follows:
Regular expressions are a set of characters usually in a particular sequence that helps find a match/pattern for a specific piece of data in a dataset.
The purpose is to allow a uniform of set characters that can be reused multiple times, based on the requirements of the user, without having to build each time.
The patterns are similar to those that you would find in Perl.
How are regular expressions built?
To start, in regular expressions, there are metacharacters, which are characters that have a special meaning. Their values are as follows:
. ^ $ * + ? { } [ ] \ | ( )
.e = All occurrences which have one “e”, and value before that e. There can be multiple e, eg ..e means check two characters before e.
^ =Check if a string starts with a particular pattern.
* = Match zero or more occurrences of a pattern, at least one of the characters can be found.
+ = Looks to match exact patterns, one or more times, and if they are not precisely equal, then nothing is returned.
? =Check if a string after ? exists in a pattern and returns it. If a value before the ? is directly beside the value after ? then returns both values.
—> e.g. t?e is the search pattern. “The” is the string. The result will return only the value e, but if the string is “te”, then it will return te, as the letters are directly beside each other.
da{2} = Check to see if a character has a set of other characters following it. E.g. sees if d has two “a” following it.
[abc] = These are the characters you are looking for in the data. Could also use [a-c] and will give you the same result. Change to uppercase to get only those with uppercase.
\ = Denoting a backslash used to escape all metacharacters, so if they need to be found in a string, they can be. Used to escape $ in a string so they can be found as a literal value.
| = This is used when you want an “or” operator in the logic, i.e. check for one or more values from a pattern, either or both can be present.
() = Looks to group pattern searches or a partial match, to see if they are together or not.
Special sequences, making it easier again
\a = Matches if the specified characters are at the start of the string been searched.
\b = Matches if the specified characters are at the beginning or the end of the string been searched.
\B = Matches if the specified characters are NOT at the beginning or the end of the string been searched.
\d = Matches any digits 0-9.
\D = Matches any character is not a digit.
\s = Matches where a string contains a whitespace character.
\S = Matches where a string contains a non-whitespace character.
\w = Matches if digits or character or _ found
\W = Matches if non-digits and or characters or _found
\z = matches if the specified characters are at the end of the string.
For further references and reading materials, please see the below websites, the last one is really useful in testing any regular expressions you would like to build:
I found this link on the internet, and would thoroughly recommend you bookmark it. It will also allow you to play around with regular expressions and test them before you put into your code, a very recommended resource Testing regular expressions
When coding in the Python language there are particular python reserved words that the system uses, which cannot be accessed as a variable or a function as the computer program uses them to perform specific tasks.
When you try to use them, the system will block it and throws out an error. Running the below code in Python
When writing your code, it is important to follow the following guidelines:
(A) Research the keywords first for the language you are writing in.
(B) Ensure that your programming language highlights keywords when used, so you can fix the issue.
(C) Setup your computer program in debug mode to highlight keywords use.
With some programs running into thousands of lines of code, with additional functions and variables, it can become harder to spot the problem, so good rigour in the initial stages of coding will help down the road any issues that you may find that need to fixed.
How would you like to present your data analytics work better?
When starting your data analytics projects, one of the critical considerations is how to present your results quickly and understandably?
Undoubtedly this is true if you are only going to look at the results yourself.
If the work you do is a repeatable process, a more robust longer-term solution needs to be applied, this is where Tkinter can help, which is a python graphical user interface.
There are many applications for using Tkinter, such as:
Use them to build calculators.
They can show graphs and bar charts.
Show graphics on a screen.
Validate user input.
Where this all fits in with data analytics?
While going through a set of data and getting some meaning to it can be challenging, using the python graphical user interface tutorial below can help build the screens that will allow a repeatable process to display in a meaningful way.
Ultimately, you could do the following:
Build a screen that shows data analytics errors in a data set, e.g. The number of blank column values in a dataset.
Another application is to run your analytics to show the results on a screen that can be printed or exported.
Similarly, you could also have a screen where a user selects several parameters that are fed into the data analytics code and produces information for the user to analyse.
There are many more ways that you could do this, but one of the most important things is that data analytics can be built into a windows environment using Tkinter that the user would be used to seeing. As a result, this could help to distribute a solution across an enterprise to lots of different users.
The only thing that needs to happen is that the requirements the user needs are defined, and the developer then builds on those, with the data analytics code run in the background of this program with Tkinter and output into a user-friendly screen for review.
It will demonstrate how a Combobox can be used to select values and then validate the entry chosen.
Using a Combobox in the computer programming world has been around for some time.
It is a useful way to select from a choice and could in many ways in data analytics help as the following examples show:
Select a date to filter a data set down to values that are in the dataset.
Using matplotlib to plot data points in charts, you could have dynamic values that change the diagram based on values chosen from the Combobox.
Utilizing data analytics reports that the user accesses, the Combobox could be used to change the data shown dynamically to allow comparisons.
When looking to fix data quality issues, use the Combobox to select values for a date that needs to be fixed, apply the fixes on screen, and then save back to the database.
Developing a Tkinter GUI and the possibilities it brings
In this video, we use ttk, written to help split the behavior of code from the code implementing its appearance.
You can see plenty more on it here ttk information. This is a handy piece of functionality as styling an object can interfere with how it works.
We also have a function that helps with the validation. In the below, it accomplishes the following:
Allows the combobox value selected to be retrieved.
Validates the entry chosen in the combobox using an if statement.
def checkifireland ():
x = combolist.get() # asssigns the value inside the combobox to x so it can be processed
if x == "Ireland":
messagebox.showinfo("Correct answer", "You will love it in Ireland")
else:
messagebox.showinfo("Incorrect answer", "You should visit Ireland first!")
The effectiveness is especially handy as it helps to ensure that the code returned from the Combobox to the function is correct
The below video will take through this step by step and explain the concepts discussed above.
Let’s make the introductions 🙂 Tkinter is a package that allows a programmer to build a GUI interface, which then can be opened on a computer screen by a user. There are many different types of GUI apps, but examples include a calculator or a text editor that opens when you click it.
Tkinter would be the most commonly used GUI package in Python, due to its simplicity, but PySimpleGUI, PYQt or PySide are other alternatives. Ensure you research these before using to make sure they suitable for your needs.
Why use Tkinter?
Relatively simple and easy to learn, upskilling is quick.
A great introduction to the concepts and ideas for building GUI apps, you will get a good grounding in the techniques and approaches needed.
Very well documented, so a programmer should be able to find the answer to anything specific they need to understand.
Now we are introduced, let’s see how to utilise it:
Install Python as usual, and make sure that Tkinter is working and you have the correct version. Note that import tkinter is for version 3.x, before that use import Tkinter
Please note that you will see in places when using Python code, that capitalization is important. This will sometimes puzzle you as to why some of your code does not work, usually, the interpreter should flag it for you.
While it is correct to put a capital at the start of a line, the programming language will ignore written English convention. An example is as follows:
list = ['a','b','c']
Print(list)
gives
NameError: name 'Print' is not defined
whereas
list = ['a','b','c']
print(list)
gives ['a', 'b', 'c']
No errors
When saving your python script DO NOT call it tkinter.py as I did, the import statement will not work. Call it something like tkinter_test.py, see red arrow below.
At the start of the video below the code will look like this. the first six lines are the creation of the Tkinter screen its size and any buttons that will appear on it.
Note that all code should appear betwee line two and line six, so that the screen output works and looks correctly.
Added to this code in the following video:
Button – which will open our YouTube channel
An image
A clickable link – Which will bring you to our Home Page
In Data Analytics, that is a very relevant question, and something I look to implement in most projects, sometimes it is too easy to click the shortcut icon to your spreadsheet application!
Changing a file is the natural step; tracking those changes are just as important.
Change is part and parcel of life, but in the technology world with the complexity and interdependency of systems, not effectively been able to track what goes on leads to:
Countless hours are trying to figure out where it went wrong.
You do not understand what needs fixing.
Systems/processes that ultimately work seamlessly, slow down unnecessarily.
In data analysis, as the volumes can be quite large, the human cannot feasibly review a set of data and find out where the underlying problem is. Well, they can, but it would take so long, nothing else would get done. This article by Forbes – predictions-about-data-in-2020-and-the-coming-decade predicts the consumption of data will just be getting bigger.
Let the script work, see the log of changes in the output.
How do you remove characters from an imported CSV file, looked at some data cleansing techniques, but there was no way of knowing what was changed other than a visual inspection. Here we introduce the data set into a data frame, change some of the values and show the output on the screen. But more importantly, as we progress through these steps, we are saving the changes as we go along.
The reality is that in large corporate settings, visual inspections would take up too much time and resources. An IT solution to help with giving the vital information required will reduce the data errors happening and allow for a more unobstructed view of how the companies data has changed over time.
Where does the trail lead to next?
Changes made to data needs a clear way of being able to be tracked.
How you captured those changes on your systems, needs to be addressed.
Implementing better systems will help you have confidence in your data changes.
This website uses cookies to improve your experience. We'll assume you're ok with this, but you can opt-out if you wish. Cookie settingsACCEPT
Privacy & Cookies Policy
Privacy Overview
This website uses cookies to improve your experience while you navigate through the website. Out of these cookies, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may have an effect on your browsing experience.
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
