Why do you want a new job?

Top Ten Reasons Why Large Companies Fail To Keep Their Best Talent

Is it one of these reasons? I've defiantly changed jobs because of some of these complaints. Developers in the BI area have somewhat of a green field of jobs to choose from. Companies need to recognize the retention risks around keeping their most talented employees.

The Downside of Stored Procedure Based Query Items

In Cognos Framework Manage one of the query item options is a stored procedure. This item seems promising, but I would suggest avoiding it. First of all, to setup a stored procedure based query item in framework manager:

1. Write your stored procedure. It can expect prompts.
2. Select that stored procedure in Framework Manager as a query item source. (I would suggest pulling this in at the database level, or at the business level if you want to make it clear that it's not a base table) It will be scanned and expected prompts will be presented, similar to a drill through definityion.
3. Enter a prompt macro in the 'Value' field. At the report level we can setup a prompt to populate this value. This procedure expects 1 prompt of 'dept'. We can define a parameter of dept in a report to pass this value.

Within the framework model, this will behave as an any other query item. It can be related to other items in the model.

The major downside to this is in the execution plans cognos gennerates around these stored procedures. In a normal query cognos' query engine will look at the selected objects and gennerate an optimal query based on the relationships defined in the framework. Ideally, all the query work can be gennerated in 1 query that can be sent to the database. In the case of stored procedures, since cognos doesn't know about the contents of the stored procedure, the sql in the procedure is not included in the optimization plan. Meaning local processing will be required to join the results of the stored procedure to any other object in the model. Be careful to only include absolutely necessary queries in stored procedures.

I would use stored procedures for any procedure needing to use several temporary tables or user defined functions. (Although, in the case of user defined functions, I'd rather registered them with the Cognos gateway so that they can be called directly from Cognos reports)

Display a # of rows per page based on output type

Most of these articles are about things that frustrated me for enough of my day to swear a bunch and then say 'hey that's annoying enough to make an article about'. Today's tip has the added bonus of being part of a report I didn't even want to build. My user wants a summary of dollars over a variable period of time with the ability to drill into the total and see the distinct set of users that gave those dollars. This is fine, but my dataset has some spots in it that it'd be pretty inefficient to drill through on. Though the standard deviation of donors/unit is pretty high, I'd rather work on a way to assure a horrid query is never passed to the database. (hard to read, large query, resources, developers headaches, etc. But hey data analysis for potential hazard queries and allowing dump reports are post to save for another time)

That being said, the user also wanted to potentially run this thing out to HTML or Excel. Well, if the data set is potentially large I want to show a good amount of it on the screen in html but not enough that rendering will take a significant amount of time. For this you can use the Rows Per Page property.

This is fine for HTML only. Based on this setting when you try to run to xls 2007 this rows per page option will create a separate workbook tab for each 150 rows. If you run to xls 2003 single sheet cognos will contain the output to one workbook but the headers and footers will be repeated each 150 rows. BUT, in xls 2007 if the default rows per page (20 unless your admin changes it) setting will cause output to be rendered on a single tab.

What do we do? I want 150 rows in html and 1 workbook tab in XLS. Well, it's Report Expressions to the rescue! There is a cognos report function ReportOutput() that will return a string populated with the output type at render time. The function definition (as of the 8.4.1 documentation) is defined as:

ReportOutput ()

Returns the name of the output format. Possible return values are: CSV, HTML, layoutDataXML, MHT, PDF, rawXML, singleXLS, spreadsheetML, XLS, XML, XLWA.

Which is somewhat obtuse, but here's a nice chart of the output formats that I'm interested in. One thing to note is that you need the right string for the version of xls you're expecting to use:

CSV	Comma Seperated Value
HTML	HTML output
layoutDataXML	XML
MHT	I assume this is a MHTML output type for 'saved html'
PDF	PDF Output
rawXML	XML
singleXLS	Excel 2000 single sheet
spreadsheetML	Excel 2007 output
XLS	appears not to be generated in xls outputs as of 8.4
XML	XML
XLWA	Excel 2003

So, ideally I'd like to be able to update the properties setting on the list based on the report format. I couldn't figure out if that was possible to overload via javascript, so I took the easy way out. I created 2 lists, one that renders when xls is run and one for html:

1. Create a copy of your list A and A1.
2. Set the rows per page property to 150 on A.
3. Leave the rows per page property blank on A2
4. Create a new string variable with 2 Values 'XLS' and 'Other' the expression can be defined as:
   

   if (ReportOutput () = 'spreadsheetML') then ('XLS') else ('Other')

5. Set the Render Variable property to this new variable on list A and check the Other value.
6. Set the Render Variable property to this new variable on list A1 and check the XLS value.

Interviewing for a BI position

I've found that interviewing for a position in a business intelligence department is an infuriating dance of half truths, hope and plans. Unlike traditional software development or IT, your experience in business intelligence seems far more dependent on the team members and department around you. That's not the interesting statement from this post, that should be self evident. Business intelligence departments concern themselves with bringing IT knowledge onto business's turf. We're not working on traditional (or even extremely quantifiable) technology products, like say, pushing out exchange server because it's better for the enterprise than that old sendmail box. We are embedding ourselves in the business unit we are working for. Discovering their process and supplying them materials to enhance decision making in that process or at times shedding light on inefficiencies in that process.

A big indicator of weather you can achieve that goal is measured in the BI world by the meta-metric 'BI Maturity'. In the past, where I've seen it, BI maturity has been calculated with some things like, turnaround time on projects, size of userbase, or other such numbers. Those metrics don't tell me how easy it will be to work in that department, projects can be short or long and failed, the userbase could be huge but only a minority percentage of the available community, or worse they could be users that never login. Worst of all they don't tell me how effective the department is in the community. Does finance hate the IT department? Does HR think BI is junk? I want to know that! I want to know how easy it will be to work in that department, these rankings of maturity don't really answer the main question that I've got when I'm interviewing for a BI position: "How effective is this department, and how interested are it's customers in analytics?".

So like any data nerd, I came up with my own BI Maturity scale. Rank the company in terms of the following categories (in no real order of importance):

While I'm interviewing I kind of keep a tally of these areas in my head and try to ask some questions to fill out the numbers.

1. Leadership - The knoledge level and direction of the department/team leaders. The ability of the leadership level to accomplish their stated goals.
• Possible Questions: 
• What's the 1 and 5 year direction of BI here? What do you see the team looking like then?
• How do you pick projects to work on? What is expected from a proposal point of view?
• Describe a project where the customer wasn't meeting their end of the relationship. What did you do to remedy that project? (Here I'm looking for a leadership level that puts up effective exit points for projects rather than one that say they can save every project. Sometimes the best use of time is to come back later)
•  What would you like to do that you currently aren't doing? What successful projects needed stewardship and how did that occur?
• Do you have a system of mentoring?
• Is there an enterprise mandate for BI or are we off on our own?
• Are there several smaller BI teams within each department that may be working on their own projects?
• Describe the roles of users at the university. What BI tools do these roles each use? (analyitics, reports, query, dashboard, etc) What types of people are in these roles?
2. Staff - The experience and skill level of the staff of the department/team, these would be your direct co workers for a non-leadership position.
• Possible Questions:
• Of the senior resources, what are there past experiences on previous successful and unsuccessful projects? (I'm looking for team members with BI specific experience, has your dba done dba tasks for BI specifically? Is your etl developer an etl developer or a transactional programmer that knows sql)
• What is the ratio of senior (in experience) employees to junior?
• Who would I get to mentor? Who would be my mentor?
• Describe the project lifecycle of a previous, successful project, what phases of development occured and what was the engagement like? (Here I am looking for them to describe a mature design, functional requirements gathering, and Testing process. You know, the stuff bad departments skip. I'm looking for them not to say the words 'excel layouts were given to the developer', but that may be asking too much.) 
• What development methodologies do we practice? What standard tools or processes do we use? Do we use version control? Do we use a robust change management process?
• Do we manage our own hardware/servers? Can we tune them how we want?
• What is the restore plan?
3. Money - Does the department have resources to do BI? Are they struggling to make do with a specific budget.
   • Possible Questions
   • What software do we use? Is there a cordinated, enterprise software purchase plan?
   • How often do team members get new hardware
   • What training or confrences did team members last attend?
   • Do we manage our own purchasing or is it through a central IT structure.
     
4. Business Buy-In - What's the excitement/need level of the business users for BI? Are they clamoring for it and have ideas of what they want and pain points to address? Are they going to be engaged fully in the projects?
• Possible Questions
• Describe a successful engagement on a prior project from start to finish. How was it proposed, how were requirements gathered, how was it deployed and maintained. (I'm looking for a lot of answers here about how the business is very involved, wants this stuff and is driving a lot of the requirements.)
• Describe an unsuccessful engagement and how it was mitigated or remedied?
• What projects are in the pipeline? Is customer demand there?
• How do they like the software we push out? Are we trying to supplement what they already have or replace something? 
• How willing are the business users to be involved in the development of their applications. (This is a big one, mature BI departments realize that this has to occur. Resources need to be full time on this thing from both sides through all stages, design, development, testing, support. Non-mature BI departments will look at these applications as deliverables, IT projects that can be built and delivered to a business customer)
• Can the users develop their own content? Are the willing to? Describe a project where this occurred or is occurring and how that's maintained.

So Simple. What's the scale? I dunno, lets say 10/10 is the highest, but the actual numbers are arbitrary. If you ask questions about the department around these areas you'll end up with a good picture of the effectiveness of the department.

Now, the thing for you to think about is where you want to a work. A less mature BI department may have it's pluses, you might be getting in on the ground floor and build some good practices and clients. A mature department may have it's minuses, everything could be so prossessified that your role would be robotic and limited in scope. (You could end up the framework modeler for specifically the database source layer!)

Cognos, get with the Program(ming!)

Here's is the current browser support list for IBM Cognos 10 Fix Pack 1 (The latest list is here):

Web Browser Support by Component
Cognos Connection / Report Viewer / Query Studio / Metric Studio	Operating System	Support Status
Microsoft Internet Explorer 8	Windows	Active
Microsoft Internet Explorer 7	Windows	Active
Microsoft Internet Explorer 6 SP1	Windows 2000 / 2003	Not Supported
Microsoft Internet Explorer 6 SP2	Windows XP	Compatible
Firefox 3.6	Windows, OS X, UNIX, Linux	Compatible
Firefox 3.5	Windows, OS X, UNIX, Linux	Active
Note: Microsoft Office integration is only supported on Windows platforms (i.e. export to Excel)
Report Studio / Analysis Studio / Event Studio	Operating System	Support Status
Firefox 3.6 (12)	Windows, OS X, UNIX	Active
Firefox 3.5 (12)	Windows, OS X, UNIX	Compatible
Microsoft Internet Explorer 8	Windows	Active
Microsoft Internet Explorer 7	Windows	Compatible
Microsoft Internet Explorer 6 SP1	Windows 2003	Not Supported
Microsoft Internet Explorer 6 SP2	Windows XP	Compatible
Business Insight / Business Insight Advanced / Active Report (13)	Operating System	Support Status
Microsoft Internet Explorer 8	Windows	Active
Microsoft Internet Explorer 7	Windows	Compatible
Firefox 3.6	Linux, Windows, UNIX	Active
Firefox 3.5	Linux, Windows, UNIX	Active

 Notice anything odd? It's 2011! These browsers were out of date when IBM Cognos 10 was released. And it's only going to get worse.

IE 8 was released in 2009
Firefox 3.5 was released in 2009 (they are currently on version 6, but it may have changed while I am typing this)
Chrome has never been supported


That might have been fine in 2009, but the landscape of browser use has changed significantly since then:

Source: StatCounter Global Stats - Browser Market Share


So now, when I roll out a new datamart I have to convince users to downgrade their browsers. Aside from usage patterns, I'm already testing my reports in multiple browsers for our user community (my users can install any browser that they want), a greater focus on keeping compatibility with them up to date would likely lead to a more robust non-ie experience. In discussions with my IBM rep they are 'looking into' support for Fire Fox 4 and won't commit to any versions after that. This support model doesn't seem tenable. At the very least I'd like the save icon to work in Fire Fox 4! (it currently does not!) And don't get me started on GoMobile and tablet/mobile browsers.

As many bloggers have noticed already, browser software development is becoming so fluid that it has transcended software versioning altogether. I'm not sure if I totally agree with the voracity of that statement, you'll probably be interested in the version when your 3rd party app stops working, but the point is valid. The current breed of web browsers Chrome, Fire Fox and even IE are continuously updating. Auto updating is now the default behavior. And use of these browsers continues to shift.

If a vendor leverages a web browser's in it's product, forcing a version number is no longer a safe harbor for you to get out of support. IT departments are either going to have to enforce a particular browser version forever, IBM will have to develop it's own application or it's just going to have to bite the bullet and continuously updates for these new versions of browsers. And they should! I want to use the latest Javascript debugging tools in Firefox and get the new security updates.

List Prompts for Dimensional Reports

If you are writing a dimensional report and prompting on particular member, you'd usually present this in a tree prompt. But in the case that you want to guide the prompting options a bit or if the dimension is very simple, this format might be confusing to users. For example, I have a filtering dimension called 'Annual Fund Indicator' with 1 level and 2 values ('Y' and 'N'). So I want to slice my cube based on some set of these values. The easiest way would be to use a tree prompt and allow the usser to select any combination of the flag.

The user can select 1 value, or both. There are some functional drawbacks to this though. For a complex hierarchy, a tree prompt makes sense. Users can navigate the tree and select the members that make most sense to the report. But with a simple set of selectable items, the tree becomes cumbersome administrative overhead.

• This is the presentation of members at 1 level of a tree with no children. Users are confused that the tree prompt has the expansion + option but nothing below it
• Users are confused about the wording on the prompt and can be confused about the lexical rules for slicing both Y and N values (they would have to select both) or 'Y' but not 'N'
• Here the member captions are shown as values, it may be confusing to users to translate the code 'Y' to 'Transactions that have the annual fund indicator set to yes' or more functionally 'Yes, Include Annual Fund amounts'

So for these, we should use a list prompt like this.

When you select a member from a tree prompt, the value used for the parameter is a Member Unique Name (MUN). We still want each of these options to resolve to MUN addresses, but now we'll have to construct them ourselves. For the bottom 2 options 'Non Annual Fund Transactions' and 'Annual Fund transactions' I created a query to select the MUN and a Caption as the Use and Display values for this prompt.

Annual Fund Include Indicator - [Cube].[Annual Fund].[Annual Fund Include Indicator], this expression is a level so will return each member of that level when the query executes. This will be the Use Value of the prompt.

Annual Fund Label - So, I want a specialized label for each member, to do that I need to get a string value of each member to run through my case statement. This case statement uses the roleValue() and currentMember() dimensional functions to get a string value of the business key of the current member and creates an appropriate caption. This will be the Display Value of the prompt.

roleValue('expression', member) - expression can be any of these: 

'_businessKey', '_memberCaption', '_memberDescription', '_memberUniqueName' and will return a string value of the expression.
 currentMember(hierarchy) - returns the current member of the specified hierarchy, since we are traversing a level at runtime for Annual Fund Include Indicator, the current member will be each member of the level.

So that gets me the MUN and caption for those values. But I want a 3rd option for the 'Both Annual Fund and Non-Annual Fund Transactions' I created a static choice in the list with the MUN for the top of the Annual Fund Indicator hierarchy. So this will slice me both 'Y' and 'N' values when I select it (as they are children of the top of the hierarchy).

So when all this is said and done we end up with a list prompt that behaves like a complex tree prompt and I use it in my query like any other MUN filter:

[CUBE].[Annual Fund].[Annual Fund].[Annual Fund]->?Annual Fund?

I did this in IBM Cognos 8.4.1 but would work the same way in Cognos 10.

Change Management: Project Release Numbering

In a previous post I said I'd post something about my numbering scheme for release candidates. If you recall from the previous post, a release candidate is the set of code that will be moved from development to a higher environment. They are complete enough that by packaging and releasing them we are saying 'this set of items can move into production'. This is the key to change management in these environments. If you are working in an Agile environment you may have a release every iteration, if not you may be releasing things more infrequently than that. This system is meant to be simple and fit with the change management process defined in the earlier post.

Any version numbering scheme should be developed to fit your change management process. An initial version of a project will be Version 1. Changes after that are tracked using our numbering system below. A change will increment either the major or minor revision number (very loosely and very purposefully defined as 'depending on the impact of that change') as follows;

• A Version Number is incremented when a project is wholly changed or rebuilt.
• A major change to a version will increment the Major Revision Number.
• A minor change to a version will increment the Minor Revision Number.
• A Patch can be applied if a change to an existing version is required (rather than a rebuild and increment). Additional patches increment the Patch Number.

Any specific release can be recreated by starting with that version's code sequentially applying all patches.

*Note, In my environment I don't need release things to users before they are production ready. If you you wish to denote that something being released to customers is in a pre-production state, I would call that a version 0, use the numbering scheme above for changes to that product and attach a greek alphabet prefix (Alpha, Beta, etc) to each release version in this pre production state.

See Also:

• Agile BI - Ralph Hughes

Do you do mentoring in your BI department?

I came across an interesting article on types of mentorship in this twitter post:

Keeping Great People with Three Kinds of Mentors - Anthony Tjan - Harvard Business Review: http://t.co/PbQv3bg via @AddThisless than a minute ago via Tweet Button Favorite Retweet ReplyIan Di Tullio
ianditullio

Do you do mentoring in BI? I am very much in favor of mentoring. The secondary point of this article is almost more interesting to me,

One of the most critical elements in retaining great people is effective mentoring. 

The community of BI developers is small, and the community of great BI developers is even smaller. If you come across one, hire them or work for them! Mentoring can be a great retention method for talented employees. Simply put, good people want to work for more experienced good people.

See Also:

•  Keeping Great People with Three Kinds of Mentors

Back from Tufte

I spent the day yesterday attending Dr. Edward Tufte's excellent course/seminar/inspirational speech "Presenting Data and Information". Attend this course if you get a chance.

Making a presentation is a moral act as well as an ethical and intellectual act.

You'd do well to remember that as a BI developer. EVERYTHING you develop is a presentation. A report, a database table, a fancy dashboard, are all presentations and all tell a story. They become an attempt at concert between what you built (the system), and the user. Now that we've elevated reporting to a moral level, Dr. Tufte urges us to address the content ('do whatever it takes to show the content') with the moral weight of the previous statement. Therefore,

The presentation stands and falls on the quality, relevance, and integrity of the content.

Dr. Tufte is an engaging speaker and I came away with pages of quotes and notes to share. But it seems the blogosphere has done it already (also it seems this course hasn't changed much in a year), check out this post for most of the gems I wrote down during this course. So I can quote ET quoting TS Elliott, "Talent imitates, but genius steals".

A day with edward Tufte

Tufte Design Principals

Change Management In Cognos BI

Change management is a difficult beast in any software environment, but in a multi-server Cognos BI environment it's especially hard. Partly because IBM Cognos doesn't seem to think anyone would have a multi server environment or do change management. But even a small team can quickly end up with many developers working with many different objects at different layers of the application stack (Database, ETL, Metadata modeling, OLAP objects, Reporting) and this can become a nightmare to wrangle. I'll discuss change management within a multi-developer project later, lets keep it at the package level for now. you have a multi server architecture, lets say it looks like this:

Development > (deploys to) Test > UAT/Regression Testing > Production

So how do you coordinate multiple developers working on multiple projects at different stages of release? One way we make this easier is with the concept of packages. Whatever you develop in development can't make it out of development until it is packaged into a release candidate. This packaging could be merely conceptual or you could actually move files around, but a package is generally defined as the set of data, metadata and bi objects for this application. (this includes all database changes, etls, OLAP objects, scripts, etc for that version of the application to function in production) This candidate can then move through the other environments and eventually into production, but only that package. Any failures in the testing stages, for example, would need to be addressed in Dev and promoted as a new package. This also lets me know what's in what environment as I can refer to them as packages (HR v1.2.1 is moving to test, Accounting v1.0 is moving to production, etc). We do some other neat tricks for version management with subversion tagging, documentation and a change control software for Cognos items called Motio, but this is the heart of my change management procedure. Only whole packages can move out of development or into the next stage! Here's what that workflow might look like.

In my next post I'll talk about the numbering scheme I use for packaging. Hold on to your seats!

thanks blogosphere

Sometimes Cognos fixes something and doesn't let me know about it. And sometimes those questions come around again and I google for them and the BI Bloggosphere helps me out. Today I had a user who wanted me to change the default analysis of a cube. Thanks guys.

My BI Experience: Analysis Studio Default Analysis report: "Recently had a requirement where a blank Analysis Studio report needs to open when you open an existing report and click on new report icon"

Debugging Cognos Javascript

How do you usually debug your Cognos reports with embedded JavaScript? Mostly I develop in IE and then load whatever I'm working on up in Firefox and use the Firebug plugin to look for any errors. (mostly because Firefox 5 doesn't work for report studio (the menu buttons don't work?)). I think I'll put together a longer post on using firebug to debug Cognos issues, but does anyone use any other methods?

Oh, hey, there's my typo.

Visual Analyitics in the NYT!

Some Visual Arts students in Manhattan graphically represented some data about their lives in the NYT today. Pretty cool analytical design.


http://www.nytimes.com/interactive/2011/07/24/education/edlife/20110724_edlife.html?ref=edlife

Working with Dynamic Filtering Part 2

If we recall, dynamic filtering is a Cognos function allowing us to limit a general target report with contextual information from the drill source. Filter the target by the source.  Like this, drilling from an Analysis intersection to a details report.

In the previous segment we went through how dynamic filtering works in Cognos 8 and 10. In this one I'll details some issues I had with the way that this works. When I initially looked into this I was working in Cognos 8.2, at that time there was one show stopper (#2) in analysis studio and a few others in Power Play Web, but they've corrected some issues as of 8.4.1 fix pack 3 and Cognos 10. Here's some of the issues I discovered with this latest attempt. 

IBM Cognos Knoledge Base

In reading some background info on Cognos dynamic drill through I got a lot of current info from IBM's knowledge base. I have to say, since they were taken over by IBM the knowledge base area has gotten a lot better. For about a year it was a barren land full of out of date information and dead links, but here's a few good ones.

http://www.ibm.com/developerworks/data/products/cognos/

Specifically the Developer Works area is nice. They have a lot of case studies and examples for some esoteric tasks.

• Member Functions
• Calculate a running total using member functions (OLAP)
• Hands on with the Dev Kit
• Working with mashup services
• Dynamic Reporting with role based security

Java Script Examples

• Restricting a Tree Prompt
• Dynamic sorting in Cognos 8

Working with Dynamic Filtering

So one question I usually get asked by users when I build an analysis object is 'How do I drill through to the details?'. As much as we want to talk users out of looking at the details when we're building them aggregates, the reporting task of detail dumps invariably comes up. It's also a likely a sign of an immature business process, but that's a whole different post.

In relational scenarios this is straight forward, a drill target mapping can be used to jump from any context to any other, map the parameters and there you go, from a summary level, select the details.

In IBM Cognos 7 this was not too difficult, you could setup an IMR details report, link it as a drill target and the dynamic filtering would filter that result set based on the context of your selection at the time of drill through. (based on business key). In Cognos 8 this process has become a hectic jumble of xml, sql generation, report variables and other black magic. Don't be fooled by the numbering, Cognos 7 and 8 are totally different pieces of software and handle the drill through process completely differently. This contextual filtering in Cognos 8 and Cognos 10 is handled by a setting you may have seen in drill through definitions called 'dynamic filtering'.

Attending the Tufte Course in Chicago.

I'm attending the Edward Tufte course coming up here pretty soon. I'm excited about this Tufte is an interesting figure in the world of data. His books deal with the presentation of data and visualization in a very smart and beautiful way. A nerd AND an artist. Hopefully I'll get some good ideas about data visualization and won't fall asleep like at most conferences.

http://www.edwardtufte.com/tufte/courses

Are you BI Curious? (Business Intelligence)

Hello, I’m a Business Intelligence developer and have been for several years. I’ll use this blog to post some best practices and interesting examples. There’s not a lot of good information out there on some techniques so hopefully this will be a good resource.

I’m currently doing a lot of projects in Cognos 8.4.1, Cognos 10 and Oracle so that will be the focus.

Here’s some pages I took inspiration from:
http://unofficialcognostraining.blogspot.com/
http://cognosandme.blogspot.com/
http://www.biprofessional.net/