Learning python for BI

Hello All

What does Python have to do with BI report? Well maybe no much directly in enterprise environment, but python can be very useful for my garage project. I want to build a Data visualization product that shows real time update of certain market driven products and get users to subscribe to it for a monthly fee, that's what python can be handy for my ETL process.

Just to give an idea, I may choose to store my data in one of these open source DBs like Hadoop Hive, Cassandra or Mongo DB, and I can use python to pull data from whatever source that i want, for example zillow.com, if i want to analyze all of the real estate properties in America. For visualization, i can use Tableau to display the market trend and my analysis..

Now you may think that my idea sucks and there are so many real estate websites out there already. Well of course the idea sucks because this is not what I am really trying to explain you here, if i had a great idea, I wouldn't be letting it out until i secure my billion dollar patent first. Duh..

Anyways, Python can be a very powerful yet quite sample programming languages that supports various DB commends (of course, much more than just doing things with DB), so it's worth your time to look into it if you are a techy guy.

Get good at it and you can write your own ETL tool.

A good place to start if you have very little experience is here:
http://learnpythonthehardway.org/book/

Thanks

Until next time

Read More

Tableau Visualization -- How to build Waterfall chart

Waterfall chart can be used in accounting to show trends in the horizontal time span usually as result of positive and negative influences in sequential order. We will look at the product's profit over years.

To start, lets have profit in row and  product and time in column:

As you can see, some product creates positive profit and some creates negative profit.

Now let's add a new table calculation off profit to get the running total of the profit:

Now the chart looks like this:

We want to distinguish the negative from the positive. To do that we create condition on profit by adding a new calculated field ('profitable ?') with the following formula: SUM([Profit])>=0

This will create a binary output: greater than 0 or less than 0

Putting 'Profitable ?' into the color windows, the chart is colored:

Now change the view from automatic to ''Gnatt" and the chart changes:

Put profit into size windows so that all of the Gnatt size will change based on the profit amount:

Now you might notice that all of the blue color Gnatt bars are growing downwards while the yellow ones are growing upwards. This is because the blue color happens to represent negative profits. Some user maybe okay with the way this chart looks, while others may prefer to flip the blue bars and make it go upwards just like the rest. To do so, let's create another calculated field call 'Negative profit' with the following formula:
-[Profit]

Put 'negative profit' into size windows and now all of the bars are facing the same direction:

Thanks

Until next time

Read More

OBIA Financial analytic AP Process 101

Today we are looking at the basic process of AP or Accounts Payable in Oracle EBS for financial analytic. This is to give you a basic understanding of how AP process works for most companies that implements Oracle EBS. In order to build AP reports out of EBS source, you need to understand the basics.

Although different companies will have different accounting process and different product dealings, the basic AP process is comparable.

Based on the below diagram, the AP process includes 4 main blocks:

Purchase Orders:

These are the purchase orders that the company has decided to execute through Oracle IProcurement. It basically means that the company has decided to buy, this can include items for the office, materials that the company products need, the contract for consultants, laptops, speakers, insurance services or electricity and gas.

PO usually consist of 3 hierarchies:
Header - highest level.
Line - under each header, there can be multiple lines.
Distribution - Under each line ordered, it can be used for different locations or cost centers. Manage A from this department uses it for a few weeks and then Manager B from another department uses it. From accounting perspective, it may belong to different cost centers

Invoices:
This is when the invoices of these PO have been entered. They have mirror the hierarchical structure as Purchase order. Each of the 3 AP Invoice tables can join to the corresponding PO tables.

Receives
Now that the company has been invoiced by vendors, they will receive goods and services that they purchase. The act of receiving is recorded in RCV_TRANSACTIONS table as transactions. Note that receives are at invoice line level. If your company makes a lot of order from one vendor and the goods will be delivered throughout a period of time continuously, you may want to check the Receipt_Date and take the Max of this date for each transaction, or it could potentially influence the grain of each invoices. As an example, most of the time you are interested in seeing how much you have totally spent in your trip to the local grocery store but not necessarily interested at each items you purchase, or at the end of a period, how many chairs does your company receive rather than how many chairs received on each day.

Hold
Hold indicates the status of your company's decision on handling the invoices. Let's say after receiving the invoices, the department realized that it doesn't have all of the goods that they thought they ordered. This could be categorized into 2 main reasons:

1. The company makes a mistake in dealing with the transactions. Maybe the goods are delivered to the wrong department or miscounted. The error lies within the company, therefore they are still responsible for making the payment on time according the to invoice.

2. The company thinks that the vendor has not completely delivered their services according to the PO. The order says 100 laptops but the company only receives 95, therefore the company puts this invoice on hold and waiting for the delivery of the remaining 5.

These information will be stored in AP_INVOICE_HOLD_DETAIL table and HOLD_TYPE will indicate what type of hold this invoice is. For each Invoice_ID, there can be multiple HOLD_LOOKUP_CODE to tell you the specific reasons for being on hold.

Now if the hold is deemed to be vendor not fulfilling its responsibility, the HOLD_RELEASE_DATE column will be used for the invoice payment due date, that is, until the error has been corrected, the hold will be released.

The payment_date and due_date are stored in various tables, but AP_PAYMENTS_ALL, which is the block at the bottom of the 4 blocks in the diagram, generally stores this information.

So for accounting purposes, the payment date and due date are based on HOLD conditions.

The process of determining what HOLD condition is, or whether there should be a hold or not, the called 3-way match, which means you have to compare the PO to invoice to the receipts to see if they matches or not.

If there is reporting requirements that needs OBIEE report that does these kind of comparisons, you should already know the logic and the type of report that will make sense.

Thank you

Until next time

Read More

Tableau Visualization -- How to build Gnatt Chart

Gnatt chart can be useful in showing status of large number of items and things with respect to time. An example of that is displaying the various flight time from each route and determine which has the longest delay or longest flight time as reflected on the chart by the size of each Gnatt.

In my example, i am going to show for a given year, how are the order to ship time is like for each product in the warehouse.

So start with product dimension (Row section) and order date (Column section):

change the order date display format to 'exact date'

Now to figure out the exact order to ship time, we simply go a date diff between order date and ship date. This is similar to timestampdiff function in SQL, but in tableau we use a different syntax by create a new calculated field called 'Time to ship' with the following formula:

DATEDIFF('day',[Order Date],[Ship Date] )

Sometimes, there are different types of shipping. So if you want to make a distinction among the dimension with certain categorization, we simply put the categorizing field into the color window. So in this case, different product has different ship mode that we want to distinguish by, so simply drop ship mode into the color section as shown below and the chart is colored.

Now move 'time to ship' field into size windows. The size of each bar will start to differ more noticeably based on the number of days differ.

We also want to include the ship date as part of the display in details for validation purpose. So we pick ship date and drop it into detail windows

Now the chart is done. We can move the mouse over any bars and the detail information will display from the details:

here you can see the time to ship for applicance is 17 days and it shows you the date ordered and the date shipped

Thanks

Until next time

Read More

Tableau Visualization -- How to build lollipop Chart

Lollipop chart is a fancy and cute chart that has become pretty popular. It is named such way because the shape of the graph consist of a bar attached to a circle, forming a shape of Lollipop. This chart can be useful to provide visualization for your measure numbers in terms of 'how long' it occupies the horizontal axis and the precious location of the measure numbers on the horizontal axis as a dot. The length of the bar is determined by the size of the measure number and the location of the dot on the axis is also determined by the same measure number. Lets take a look at this graph that I start with:

Here, I have sales from the measure which I put on the column section along with Region as the dimension. In the row I have Product, another dimension.

Now, right-click on either 'sales' field and check 'Dual-Axis':

Now chart transformed:

Notice on the left side where i highlighted, there are 2 'sales' under marks windows. Click on each of those 2 highlighted icons, it will open up marks options that you can set for the displays of each of 'sales' field.

So first one is to set the first 'sales' as bar display:

And second 'sales' as circle display

Adjust the size of each of the fields, it changes the graph into looking like a lollipop by making the size of the bar smaller and size of the circle bigger:

Now, let's create another field called profit ratio, which is calculated by taking Sales and dividing it by profit:

After that, drag and drop profit ratio into color windows:

Now we have create a Lollipop chart that gives you 2 different visualization of the sales and also displays the color divergence based on the profitability. If you move the mouse over to any point on the graph, it will display the detail numbers on the chart including regions, product, sales and profit ratios.

Thanks, until next time

Read More

Tableau Visualization -- How to build Dot chart

Dot chart is a pretty typically statistical graph where each data point on the graph are marked in a circle shape. Making dot chart in Tableau is pretty simple and standard.

Lets start with sample data set and we put 'state or province' as dimension then 'sales' and 'profit' as measures all into the row section of the report and set the display under 'Mark' as circle:

Then, drag and drop both sales and profit from rows into the highlighted vertical axis on the graph:

After this action, you will see that 'Measure name' and 'Measure value' appears on the row and column section automatically, we will put 'measure name' in to color window under 'Marks':

Now the graph becomes like the following, which we will move the 'measure value' into column section:

Now we get the following view:

Pivot the graph and we get the final view:

This view displays the sales and profit of each state on the graph separated by 2 dots in different colors

Thanks

Until next time

Read More

Tableau Visualization -- How to build Pareto chart

Pareto chart can be very good for highlighting certain items from a very large set of factors based on certain criteria, Tableau has a very powerful way of creating pareto chart.

We will start with a simple dimension (product name ) in Column and measures (sales) in rows, in order for the chart to make sense, there should be a lot of products:

Now in our particular example, we are comparing the running total and percent of total of the sales per product. So we create table calculation on sales. In this case, percent of total is the secondary type calculation after running total:

The graph changes afterwards to the following:

Now in this example, we are trying to compare the running sum of sales and total sum of sales, if when Total Sum/running is greater than 0.8, we should recognize those product item as above 80%  of all. So in this case we will create a new calculated item called In Top 80% with the following formula: RUNNING_SUM(SUM([Sales])) / TOTAL(SUM([Sales]))<= 0.8

Unlike writing in SQL where we may have to put it in a case statement, here in Tableau since the condition is expecting a binary output: yes or no, and nothing else, the above formula is a simpler way to do so. It will divide the entire set with either > 0.8 or <=0.8

Now move it over to color box under marks, this will automatically divide the entire product list into 2 colored groups based on whether the divided value is greater than or less than 0.8

like so:

Now change the fitting from normal to entire view in order to see the big picture:

Entire picture:

Now lets add a reference line in the graph by right clicking on the vertical Axis, and we set the line value at 0.8 and make it constant.

This will drop a line at the graph at around 80% area on the graph and it will stay constant:

Now, let's sort the graph by product in ascending order by sales:

Now the Pareto chart is done:

Thanks

Until next time

Read More

Tableau Visualization -- How to build thermometer chart

Thermometer is a cool visual that allows audience to visually look at different matrix that the difference can be visually indicated by the different color, size in one chart as if it was like Thermometer.I experimented a few ways to build a basic thermometer chart using tableau, this is a pretty cool visual.

1. Lets start with 2 dimensions and 2 matrix. Here I am using customer segment and region as dimension and sales and profit as matrix by putting the dimensions in column section and matrix in row section. The report will automatically look like the following:

2. Drag and drop both matrix columns into the Y (Vertical axis) of the chart:

The chart will automatically transform to the following view after dropping both matrix. You will notice that 'Measure Names' and 'Measure Values' automatically appear in the following highlighted area

3. Move 'Measure Names' to Color windows under 'Marks':

The chart will look at the following after done:

This time drop measure name under size windows:

And you get the following chart that the 2 bars stack on each other:

4. Now unstack the bars by going to analysis--> stack marks --> off

Now you get your thermo chart:

Thanks

Until next time

Read More