Sunday, November 12, 2017

Pandas Merge two Data frames based on common column values

Sometime back I had this "Combining multiple tabular data files together" situation at hand and I wrote about it then.

Now, there is another problem I have to solve.... It is similar to the above but not exactly the same. I will explain it below:-

The Problem

I have tow Data frames (tabular data) in which on is a subset of the other. That is to say second data frame (df2) is a subset of first data frame (df1).

Now, I need to merge them together based on a common column in the two data frames (df1 and df2) and also keep track of what row was in the the main data frame and not in the subset data frame. This is more like saying:
- Remove rows from two Data frames that have uncommon column value
- To find rows in one data frame but not in another
- Find rows which don't exist in another data frame by multiple columns

Any how you look at it, above is the general idea!


The Solution

I solved this by using the pandas merge() method with the following flags/parameters: "how='left'" and "indicator=True".



This created a Data frame with an extra column name: _merge that contains two keywords 'both' (row contain in both data frames - df1 and df2) and 'left_only' (row contain in only the left data frames - df2).



Example:-

df_all = df1.merge(df2.drop_duplicates(), on=['col1','col2'], how='left', indicator=True)
      


That is it!

Wednesday, November 1, 2017

QGIS - Label Multiple Polygons with one attribute text


QGIS Label group of polygons with single text fro the attribute field


Suppose you have a vector layer that contains polygons with the same attribute values as seen below.




You don't want to name each of the polygons because the values are the same, instead you just want one name for the whole group of polygon as seen below...



To achieve this, create a new attribute field in the to control either the label string is turned on/show or off/hide the labels by assigning the value of 1 or 0.

1 is for: On/Show Label
0 is for: Off/Hide Label


Now, after creating the attribute field, take note of the field name and go to the layers label property tab. Define and enable the field label as usual and under: Rendering >> Data Define >> Show Label, select the label control field you created above and save your changes.



All the polygons with the value of "0" will have there labels turned off/hidden only those with value of "1" will be on/showing.


Note: There are many ways to do this, the above is perfect if you want to have control over your label within QGIS map canvas.


That is it.
Thank you for reading.

Tuesday, October 24, 2017

Map of Nigeria States' Senatorial Districts

Map of Nigeria Senatorial Districts by State - You may also be interested in Maps of Various States and their Local Governments with electoral wards in Nigeria.

The Senate is the upper house of the Nigeria's bicameral legislature, the National Assembly of Nigeria. The National Assembly (popularly referred to as NASS) is the nation's highest legislature, whose power to make laws is summarized in chapter one, section four of the 1999 Nigerian Constitution. It consists of 109 senators: the 36 states are each divided in 3 senatorial districts each electing one senator; the Federal Capital Territory elects only one senator (Source: WikiPedia).

Here are the maps or visual representation for all the 36 states Senatorial Districts as defined by INEC.

Basically, it is this table provided by INEC listing the NAME OF SENATORIAL DISTRICTS in Nigeria that am representing in a more meaningful format as a map.

Wednesday, October 18, 2017

Domain-Specific Language (Geographic Markup Language - GML)

Hello there,
According to wikipedia: A domain-specific language (DSL) is a computer language specialized to a particular application domain. This is in contrast to a general-purpose language (GPL), which is broadly applicable across domains.

There is a wide variety of DSLs, some examples include:-
~ HTML
~ Logo for pencil-like drawing
~ Verilog and VHDL hardware description languages
~ MATLAB and GNU Octave for matrix programming
~ Mathematica, Maple and Maxima for symbolic mathematics
~ Specification and Description Language for reactive and distributed systems
~ spreadsheet formulas and macros
~ SQL for relational database queries
~ YACC grammars for creating parsers
~ regular expressions for specifying lexers
~ the Generic Eclipse Modeling System for creating diagramming languages
~ Csound for sound and music synthesis
~ GML - Geographic Markup Language
~ KML - Keyhole Markup Language

Thursday, October 12, 2017

Most profitable python programming skill in 2017 and beyond

Hello there,

This blog post will address the question:-
~ What Python skills are in demand in the freelance job market?
~ What Python skills are people/companies looking for?

I will share my python programming freelance experience since the beginning of this year 2017. Am also confident that this experience will last longer and remain relevant beyond 2017...

Python is one of the programming languages that is on the rise and there's really a demand for Python programmers out there. But, the question is what aspect or area of application is more profitable at the moment and in the near future?

Applications for Python
First of all, python is applicable in many areas of our day-to-day activities. Here is a high level overview of python application domain:
1) Web and Internet Development - Django, flask, Pyramid etc
2) Scientific and Numeric - SciPy, Pandas etc
3) Education - very good for teaching programming
4) Desktop GUIs (Graphical User Interfaces) - TkInter, wxWidgets, PyQt, Kivy etc
5) Software Development - SCons, Buildbot, Apache Gump etc
6) Network Administration

Read more on the official website...

Friday, October 6, 2017

Scraping StaticGen.com with Python

This is a short Python Web Scraping Tutorial

We will learn how to scrape data from this website StaticGen and have the data saved into a CSV file for further processing.

Let's get our hands dirty...

Prerequisite

You should already have at least some basic knowledge of the following:-
1- HTML and CSS
2- Python


Inspecting the site's HTML structure

Load the website on the browser and study the html structure. Use what ever tool or browser for this, I used Google Chrome browser and it looks like below:-




Ctrl+U to view source code





FireBug to view html code




Html Inspector (Ctrl+Shift+I)





As you may have noticed, the required dataset to be scrapped are arranged in rows and columns on the web page. This makes things easier since we have a consistent pattern to follow.

Wednesday, October 4, 2017

Dynamically generate HTML file from CSV data using Python

Hello there,

Assuming you have bunch of dataset in a CSV file and you want to dynamically add them on to a HTML file to upload on a web server so the someone on the other part part of the world could access and view it on the web. And you don't want to share the raw CSV file instead you want a to share a viewer friendly file such HTML, then this article is for you.

Below is the out of the script that collects data from a CSV file and populate a HTML file.



The script is as simple as below...

So, basically we just import the CSV module and read in the csv file. Then we used the open() method to create a html file in the working directory and write lines of standard html code in it.

import csv


csv_list = []
      
try:
    with open('registrant_data.csv', 'rU') as csvfile:
        readCSV = csv.reader(csvfile)
        for row in readCSV:
            csv_list.append(row)
 # ========= Create html file ================

    with open('../nametags8gen.html','w') as html8:
        html8.write('<html>')
        html8.write('<head><title>Title of the page...</title> <link rel="stylesheet" type="text/css" href="./nametags8gen.css"></link></head>')
        html8.write('<body>')
 # more algorithms to fetch csv records...

        html8.write('</html>')
        html8.write('</body>')
         

To generate new html with new contents, just update the CSV file and run the script.

That is it.
Thanks for following...

Sunday, September 24, 2017

Resize image dynamically when frame is maximized in wxPython

Hello,

I will work you through on how you will "Resize a wxStaticBitmap image dynamically when frame is maximized in wxPython

Here I have two StaticBitmap images placed side-by-side in box sizers. So, we want the size of the images to scale appropriately with the frame whenever it is been maximized.

The code....

import wx

class MyFrame2 ( wx.Frame ):

    def __init__( self, parent ):
        wx.Frame.__init__ ( self, parent, id = wx.ID_ANY, title = wx.EmptyString, pos = wx.DefaultPosition, size = wx.Size( 500,300 ), style = wx.DEFAULT_FRAME_STYLE|wx.TAB_TRAVERSAL )

        self.SetSizeHintsSz( wx.DefaultSize, wx.DefaultSize )

        bSizer11 = wx.BoxSizer( wx.HORIZONTAL )

        self.m_bitmap3 = wx.StaticBitmap( self, wx.ID_ANY, wx.Bitmap( u"img/im1.jpg", wx.BITMAP_TYPE_ANY ), wx.DefaultPosition, wx.DefaultSize, 0 )
        bSizer11.Add( self.m_bitmap3, 1, wx.ALL|wx.EXPAND, 5 )

        self.m_bitmap4 = wx.StaticBitmap( self, wx.ID_ANY, wx.Bitmap( u"img/im2.jpg", wx.BITMAP_TYPE_ANY ), wx.DefaultPosition, wx.DefaultSize, 0 )
        bSizer11.Add( self.m_bitmap4, 1, wx.ALL|wx.EXPAND, 5 )


        self.SetSizer( bSizer11 )
        self.Layout()

        self.Centre( wx.BOTH )

    def __del__( self ):
        pass

app = wx.App(0)
MyFrame2(None).Show()
app.MainLoop()

When you first launch the GUI it is like this....



Thursday, September 14, 2017

The Geo in PyCon Nigeria 2017

Hello,

I am privileged to be among the speakers at this year's python conference and the title of my talk is "Using Python for GIS and Remote Sensing Operations".

As I Geo aware individual, I was impressed with the organizers especially the guy behind the conference Venue page.



He did not only added geospatial technology on the page, but he also encouraged users to make use of the tool to make there live easier while locating the conference venue.

The web page is based on python Django web framework (Pinax/Symposion) and on these bases of python & GIS relationships, I decided to write this blog post.


What I did on the Venue Page

On the page there is a "Get Direction" tool, so i used it to get the direction from Abuja where I am coming from.



There and then I have a quick idea of where am going to and how long it will take me to reach there by road or air as seen above.

I also get the direction of my hotel (Presken Hotel) location to the venue and I observed there are two location of the hotel having similar name. However, based on the address provided I realized the correct on is with the name "Presken Hotel (Awolowo way)" and it is about 2km (5min drive) from the venue.



Thank God for Google map, everyone can at least do or interact with GIS without installing any heavy software.



You can find the presentation here: https://github.com/forum2k9/pycon-ng-2017

Thursday, August 31, 2017

Next Step after Learning the Basics of Python Programming

Hello there,

Today I want to share with you things you could do after completing the fundamentals in Python. There is always this dilemma on what NEXT after learning and completing the basic fundamentals of Python Programming.

I will attempt to solve this using my experience after being in similar situation few years ago.

Shortest Solution


The shortest answer will be that: its depend on what you want to use python for, learn related libraries in that niche. For example, if your in the Geospatial/GIS industry then go ahead and search PyPI for related libraries and learn them.


Long Solution

Now its time to program something useful to you in Python and put that theoretical knowledge to accomplishing something practical!

Pick any topic that interests you and learn the libraries related to it. There are many tutorials and documentation out for just about any library you wish to learn.

As you go through tutorial or documentation with code in it, type in all of the code and make it work. That's how I do it. That's how every programmer does it.


Find a project that interests you and try to solve it using Python. It could be in any of the following fields:-

~ Web and Internet Development
~ Scientific and Numeric
~ Education
~ Desktop GUIs
~ Software Development
~ Business Applications

Python Project Ideas

Here you will find list of practical projects to try out.