A Missing Profession: Information Engineering

In business intelligence it seems to me like there is a missing profession: information engineering.

Business Intelligence (BI) solutions ultimately aren't about the data that an organization has: they are about the information that the data carries. This information has to be uncovered, it has to be validated, and it has to be refined in a way that is usable.

What Information Engineering Isn't
Information is different from data. For instance, imagine a bank; we've got checking balances, transactions, and monthly trends. That's data. What does this data mean about the chance that the customer will leave the bank? What new accounts and services they might want? The chance that there are fraudulent activities associated with the account? That's information. Data is something that is clear and unambiguous; information needs to be inferred from the data available. Information ultimately is meaning and that makes it both messy and rewarding.
Information design relies on database design but isn't database design.

Paradoxically data can be wrong, or noisy, or incomplete, and still carry a lot of information. For instance I was working on customer purchasing behavior and I found a segmentation code that carried a lot of data about purchase patterns. I asked about this segmentation and found it was done over a decade ago and it was considered obsolete because it was done so long ago – even though when I investigated it was very useful. Whoever had done the segmentation in the first place had clearly done an damn good job.

Information engineering isn't software engineering. Computer programs like a web browser function by presenting data in a certain form, regardless of the content. If a web page properly follows the HTML protocols then a browser can show the page, regardless if the page is the IBM main page or a blog for a cat. This means that there are clearly right and wrong software solutions. Either the pages display or they don't, and if some pages don't display that's a bug that needs to be fixed. Information engineering doesn't have clear right and wrong but it does have better and worse answers. An information engineering answer can work – produce a number where a number needs to go – but not be very good.

What is Information Engineering?
Information, like data, like hardware, needs to be crafted, extracted, built. The end use needs to be understood and the end user accounted for. Information Engineering is usually invisible. Somebody wants a number, somebody gets a number, and if that number is any good is left to the person putting the system together.

Next up: some examples. I'm getting tired of this abstract pablum, and I'm one of the most abstract guys on the planet.

Information Engineering and Design

Why am I doing this?

We design hardware. We've been doing electrical engineering for years.

We design data streams. Database engineers have been around forever.

What this is about is designing information, engineering information. People have been doing that for a while, but there isn't any name for it, no discipline, not set of best practices.

It's not databases, it's not statistics, it's not data mining, it's something that doesn't really exist yet.

Information is to data as data is to hardware.

I should have fun trying to figure this out.