Disclaimer

All of the topics discussed here in this blog comes from my real life encounters. They serve as references for future research. All of the data, contents and information presented in my entries have been altered and edited to protect the confidentiality and privacy of the clients.

Various scenarios of designing RPD and data modeling

Find the easiest and most straightforward way of designing RPD and data models that are dynamic and robust

Countless examples of dashboard and report design cases

Making the dashboard truly interactive

The concept of Business Intelligence

The most important concept ever need to understand to implement any successful OBIEE projects

Making it easy for beginners and business users

The perfect place for beginners to learn and get educated with Oracle Business Intelligence

Thursday, June 21, 2012

OBIEE Project Case Study 1 ---- Gathering Requirements

So based on my previous article introducing this project I was implementing, this is the follow up on how I gathered requirements in this project environment. Note that requirement gathering is an on-going process that doesn't end with one phone calls or one meeting sessions. So whatever I am going to talk about here is something I did repeatedly over the course of our development.

Anyways, so I joined this new project and I spent some time understanding what the company was trying to do and how they have decided to do it as described in the previous blog entry, now it's time to get some requirements. 

Now you should understand that for most OBIEE projects, the desired product from an end user's prospective will be reports that shows them data or charts that they want to see. The users aren't typically tech gurus, so they can only describe their requirements in their language. It is our job to understand what they want and figure out what's the best for them. 

So in my case, we are getting a bunch of documents with charts and graphs as desired. There are high level reports that shows the overview of each network devices within certain areas or network groups along with measures and stats to indicate the traffic counts, throughput and utilization %, and there are detail level reports that show the detail stats of each network devices found in that area. 

So now, looking at all these charts and graphs, who will know how to build them? 

Before we even go there, we need to know what are these users going to give us. There has to have something we can work with to start. In every project out there, this can be the raw data or files from transactional systems. These will be the data that users are familiar with and yet make no sense from analytical reporting perspective. 

Therefore, the first thing to understand after looking at the requirement document is to know how to obtain data for each of the network devices. In OBIEE's term, the dimension tables for each devices like LSP, Pseudo-wire, VRF have to be loaded from somewhere to begin with. Hence, this will be the first questions to ask the business. Secondly, the measures on each of these requirement documents need to be loaded into fact tables, but from where? 

These are some basic information we need from the users so we can have something to work with. In our project, it turns out that these data are coming from reference files generated from the lab that the client set up for testing products for their customers. As a result, the files are in certain format that does not make business sense.  Having mapped each files with each dimension and fact that we identified, now the next question is how to associate these files with each other? This is important for OBIEE because without it, we can not create our data models. 

Looking at the report requirement documents, we know that they want to see in each area, how many pseudo-wires are coming through, what are the top 10 pseudo-wires in each area ranked by Throughput and etc. This tells us that we need to find ways to join Pseudo-wire dimension, Area Dimension with a fact table. So the question becomes, how to find out what Pseudo-wire is coming through what Area by looking at the raw files from the lab?

Needless to say, this is the question that the business will have to tell us. The answer we get is that they are associated by an IP_Address given in one of the reference files. This is suggesting how to join these dimensions with fact. Obviously, the next question becomes, can each Pseudo-wire belong to multiple area and vice versa? This will tell us how to relate these dimension tables, are we going to create each dimension tables or denormalize them into one bigger table? At what level should be load the fact table?

Obviously, we haven't even gotten to dealing with time dimensions yet. This is another important aspect in requirement gathering. Knowing the update frequency, historical data storage and detail of time that the report is against are some of the key questions to ask in order to design our model.

Are we going to use one dimension or several dimensions? How are we going to design the schema? If I want to design the schema that joins Router dimension with one of the file for CPU Usage fact, can I do that? What does it mean in term of technology if I am reporting on a list of router with CPU usage? Does it mean sense? Can I join Interface Dimension with the file for ethernet fact which shows measures for radio station? Does radio station even have interfaces to which it terminates? The question goes on and on. 

This is a long process of gathering requirement and interpreting requirements. It is important to start thinking in terms of the business and use the answers gotten from the business to guide the implementation design. Although what I am doing here is specific for the subject of network engineering, the same method and exercises are transferable to other OBIEE projects as well.  If you are doing financial reporting, you need to understand what the business is trying to achieve, to ask questions about how they relate their GL accounts with transactions? What are the caveats that they are facing all the time? Are they dealing with multiple currencies? How do they know which currency the transaction is used? The list goes on. You will never be short of questions if you start thinking in terms of their business. The answers you get from these questions will be your good guide in terms of translating it into OBIEE language.

One important exercise I always do is keeping tracking of all of the meeting notes and sharing them with clients, users and other project managers. Requirements always changes, therefore, keeping a record of changes and the impact analysis associated with each potential changes will be a valuable information along the way.

Until next time.

Friday, June 15, 2012

OBIEE Project Case Study 1 --- Telecom Analytics

So for the past few weeks I have been very busy working with this new project, which is challenging and very unique. It makes me look back on my career and see the special things in every project I have done. It makes me realize that every project I did can be looked at as experience gained, because just like you will never find 2 persons with the exact same face, you will never find 2 projects with the exact same setups, environments and journeys. Therefore, I have decided to talk about some of the projects I have been or currently been doing as a way to  honor the experience, I hope it serves as a learning reference as well as understanding the journeys I take to overcome my own issues and challenges.

This project is for a telecom company. What makes this project uniquely challenging is the kind of report that I have to deliver. In order to understand the purpose of this project, it's better to throw away our normal understanding of BI Projects. Here, we are not creating reports for the clients internal operation, in other words, whatever we can think of that a business needs such as Financial reporting, HR, Supply Chain, Marketing etc, can be thrown out the windows. Basically, this client doesn't use OBIEE period.

Strange isn't it? Let me tell you what we do in this project and why I call it Telecom Analytics.

In order to understand the project at a high level, we should first understand what the client is trying to achieve overall. I know this is the basic consulting 101, but you will be surprised how many consultants I worked with did not have to the awareness to think at a higher level. Using the technical skills to create reports based on the requirement is one thing, but understanding the purpose of this project is something else worth spending time on. It will make everything you do much more proactive and efficient.

Enough said here. So in this telecom business, you have to first understand how things work. Why when a person dials a number, the other person's phone rings? I won't get into the detail of how signals are traveling from radio base stations through routers to another stations. Assuming it is done by a very complex router networks, therefore when customers complaints about the quality of their services, how would you even begin the solve the problem? How would you know at which router the signals gets dropped or at which traveling route the traffic is overloaded? So this is where OBIEE comes in.

As we all know, for any data warehouse projects, it is about extracting data from sources and loading them into target DB which provides BI reports. So in this project, the source we are looking at are those routes and network devices. In other words, instead of dealing with transactions, sales, orders, GL accounts, suppliers in your normal BI project, here we are dealing with Pseudo-wires, nodes, Thresholds, Utilization %, Area etc. Are these things making business sense? It won't. Do they make sense to network engineers? They have to.

Of course the raw data coming from these routers (realistically speaking, thousands of routers) won't make any sense for reporting. It probably looks like what you see in the Matrix movies. These data have to be converted into formats that OBIEE can work with, so how is this done? This is where the concept of big data, cloud computing comes in. For those who don't know what is Big Data, feel free to look it up online. Basically, this is the method that google has been using for years to make their search engine the best in the world.

The specific tool we are using to extract data from the routers is Hadoop, which is an open-source engine based on java programs. It goes after all these routers on hourly basics or daily basics and convert the data into CSV files. From there, we have decided to use OWB or Oracle Warehouse Builder to take the CSV files and load them into another DB, which is where OBIEE will be using to build the reports. The reason it has been decided this way is simply because of the price. Now, how much computing should be done at Hadoop and how much should be done at OWB or even OBIEE will be determined completely based on the requirement and our approach.

So having described the high level flow of this project, now on to the challenges.

Here are some of the challenges this project brings:

1. Lack subject matter expert for gathering business requirements. Since network engineers have no clue about how data warehouse works and most of the project folks have very limited understanding of telecom technology. Understanding the requirement is a constant challenge.

2. We are inventing the wheel here. In other words, there is no prebuild models to follow, there is no base practices to refer to and there is no study cases to look at for building telecom reports.

3.There is no way to build a clean data model. In the ideal world, we desire a clean relationship between facts and dimensions. However here, because we are dealing with configuration data from routers and network devices, it is impossible to form star schema.

4. The way dimensions are related to each other is never straightforward. There is rarely dimensional 1 to N relationship. For example, 1 Pseudo-wire can connect to multiple nodes yet 1 node can also have multiple Pseudo-wires going in and out. It can be bi-directional or non-directional. Having such tricky relationship among these devices greatly reduces our choices when it comes to building warehouse objects. De-normalization has to be done very carefully to make sure the data are aggregated correctly

5. Performance has to be very quick. Think about there are millions of signals traveling through mobile networks every second, what kind of data volume can you imagine we are dealing with.

6. Oracle support is not available for this project. Simple, because this project is not for the internal business, there is no production environment here. Therefore, good luck charlie.

7. Deadline and requirements are non-negotiable

Having said all these, you will start wondering with a thousand question marks. Most likely you will be wondering who is coming to look at these reports and how are we releasing it if there is no production environment.

Fair enough. Just think of what we are doing like creating software. Think of it like how blizzard entertainment developed "warcraft". They will only have a QA environment, the production environment will be the home computer of us and we are the users who are complaining about bugs. It is the same thing here at our project.

The reports will be created with the intention of analyzing the router network for the mobile companies, therefore our end product will have to be made into DVDs shipped along with other hardware products to big network carriers who we are paying our phone bills to.

Hopefully, you will get some idea about how this project works at the high level.

In order to succeed in this project, having good consulting skill is very important as requirement gathering is an never ending effort. It is also required to be able to think on your feet when it comes to design by throwing away the models of BI Apps. Finally, knowing telecom is also going to be very helpful. Yet, even if you don't know, this project will make you a pretty good expert on that.

The skills, knowledge and problem solving ability developed here is completely transferable to any other industry. It is not about what you know, it is about how you can pick up new things.

In the near future, I will go through some of the things I do to make this project work. It will cover functional requirement gathering to technical implementation of OBIEE as well as general consulting scenarios.

Stay tuned!




Related Posts Plugin for WordPress, Blogger...