Tuesday, January 26, 2021

Data Analytics Ireland

Data Analytics and Video Tutorials

Learn python online

Quickly learn the program through the internet or a web browser.

Python Data Cleansing 

How to remove unwanted characters

admin 174 Views 0 Comments , , , , ,

Estimated reading time: 2 minutes

Removing the unwanted, that is holding you up.
A situation has arisen of you having information, which has erroneous data inside it, what do you do?

Data issues are a common scenario faced by many data analytics professionals and the industry as a whole. Data quality now has become more critical, especially as we move more processes online and the digital landscape increases.

Most pieces of data go through a process of been transferred somewhere between systems to be used or reports rely on the accuracy of them. If the data in the source system has quality issues, the problem if not addressed before going somewhere else can then push the data quality issues more throughout an organisation, like a spiders web it expands further.

The next step, looking to fix the problem and planning for it.

To combat this data, professionals need to come up with a plan on how to tackle this, either:

  • Fix at source
  • Take the data in before moving it on, and investigate the problems.
  •  Reject the file or part thereof.

All three options above have scenarios around them with costs and implications, depending on the industry, you need to pick the most appropriate way to handle. As an example, in the banking industry payment files can sometimes have data in them that is rejected entirely or in part. But the bank may decide they will only discard the records with the wrong data and process everything else.

How to go about it and how regular expressions can help

In this video, we look to go through an example of how to cleanse a data set;

(A) We use a list to check what problems we need to find.

(B) Using functions again to process through the data to find the problem and extract them.

(C) Regular expressions also appear as they look to find the special characters in the data set.

The concept of regular expressions is used extensively across several programming languages; it is a good way to test data and find erroneous values. If you are thinking about machine learning, it is quite important to get a more thorough knowledge of how they work. Here is a good link for further reading if you need more information Regular Expression how to

Thanks for watching and if you like, please share and subscribe through the buttons on this page!

Data Analytics Ireland

Python Dataframe 

hide a column from a data frame

admin 221 Views 0 Comments , , , , ,

Estimated reading time: 2 minutes

They say there is nowhere to hide, we disagree!
As an addition to How to add a column to a dataframe would you like to learn to go and hide it?! This video has several steps in it; following each one will give you a good introduction.

To start why you would like to hide a column?

  • You may not want to reveal its output as it is sensitive information.
  • The data in the column is not in the correct format, you will want to repurpose it, so it is the way you want it.
  •  The column could be a calculated column. Hence it serves as an intermediary step before your data frame is output.

Finding the best way to hide unwanted data:

In this video, we introduce several concepts to help not show a column:

  • Specify the actual columns you want to include in the data frame, by default doing this you are excluding the column or columns you don’t want to see.
  •  We use drop, to explicitly tell the data frame not to show a particular column.
  •  Also, we display a scenario whereby you have a calculated column but do not want to show its output, based on one of the reasons outlined above.
  • Finally, the index of the column can appear in the output, so we have shown through set_index how to hide it from what is displayed.

This latest in the Python Dataframe series looks to build on the knowledge in the previous examples. We hope as you learn python online, it will increase your programming skills.

Thanks for watching and don’t forget to like and share through our social media buttons to the right of this page.

Data Analytics Ireland

Python Lists Python Tutorial 

Python Tutorial: How to sort lists

admin 171 Views 0 Comments , , , , ,

Estimated reading time: 2 minutes

Following on from our post on how to use Python lists have you ever wondered how to sort lists for your Python project?

Our latest video on lists will go through some of the techniques available so that you can get an idea of how to structure your data and sort.

Getting to understand how to implement

In this latest video we will look at:

  • sort() method
  • sorted() function
  • sorting a list through a function

 

Adding in those extra bits to help make the process smoother

Have you thought about sorting ascending/descending?

  • There is also a discussion on this topic as well, and while an index is available for the list, which you may feel does not merit sorting, there could be other logical reasons to implement sorting.
  • Leaving out the reverse = True/False in the sorted method can have an impact, though if you require it left out of the list you have created, automatic ascending will be the default.

On this channel, we have discussed a number of different ways to manage your data. In thinking about sorting a list, why would you want to do this?

Some common reasons are:

  • To visually see if there are duplicates, either on the screen or printed out.
  • If other objects are dependant on the list, say a combo box, then having duplicates visible can help to reduce the size of their contents.
  • Iteration – If you are looking to iterate over a list, it will be quicker if it is sorted.

If you want to learn about lists, using them, and how how they can be iterated over, why not visit Data Analytics Ireland YouTube channel, there are lots of videos there that will help explain the concepts discussed here further.

To get some more links on this topic click here python sort method, it is a blog posting from our website that has some useful links and explanations for you.

Python Dataframe Python Tutorial 

Python Tutorial: Add a column to a data frame

admin 165 Views 0 Comments , , , , , ,

Estimated reading time: 1 minute

You have learnt addition, now learn how to add a column to a data frame!
In our last post on what are Python data frames, we introduced the concept,  but are you now searching how to add a column to a data frame?To start, I was working away and wondering how I could accomplish this, as there were many posts about it. Searching through the jungles of website articles, some topics of interest that gave me ideas where as follows:(A)List comprehension.(B) Lambda.(C) Numpy.

Having tested the waters to see how you can approach:

After working through the above to:

  • Figure out how to use them.
  • Write some code to see how it all comes together.

The best thing to do was to put code into action!

Python for beginners or advanced programmers does not have to be hard!

Remember to subscribe to our channel if you like what we do!

Support is on the way:

TutorialsPoint is an excellent resource if looking to understand some other examples see this post here: TutorialsPoint: Add Column

To see a related post how to hide a column from a data frame look no further How to hide a column from a data frame

Data Analytics Ireland