Sunday, April 13, 2008

LTV at LTC: The LTC IT Department

During this same time we were going through a long series of meetings with the LTC IT department. The motto of the LTC-IT was “we will give you anything you want, just tell us what columns you need in your flat file”. For instance, the LTC-IT project documentation had extensive sections for listing data elements extracted and the databases they were extracted from – but only a minimal project memo section to describe what to do with those data elements. A project that didn't involve extracting data into a file was almost impossible to describe using the IT project documentation.

The LTC-IT department and my group had a very contentious relationship from the start. For instance, the LTC-IT was maintaining a marketing data warehouse, but they refused to allow marketing employees to access the data warehouse. We had to fill out data requests that a small group of data pullers would fulfill. Quite quickly my group found a back door into our own data warehouse simply to do our jobs.

The organizational interface between LTC-IT and the rest of the company was a Project Management Office – the PMO. PMOs were in theory supposed to have neither an understanding of the business nor an understanding of technical details but were supposed to facilitate communication between the business and technical side. In practice, because the PMOs wrote the project documents before technical IT got involved they ended up making critical technical design decisions and setting business objectives.

The LTV project was going to be unique for LTC. Employees in the marketing department were going to be developing formulas and code for the IT department to implement instead of giving the IT department high-level business concepts for design and implementation. The project manager (PM) and I spent a number of months working out the details of the interaction between our departments. When the project got to upper PMO management it was soundly rejected. According to the PMO office the LTC-IT did not have the technical competence to support model implementation and that the Marketing Department would have to supply the technical expertise to implement the LTV models.

I was overjoyed at this news. It meant minimal contact with the LTC-IT and PMO and that I could have direct oversight over the most critical matters.

The next issue we had to resolve with the LTC-IT department was where to run the LTV system. If we were going to be putting scores into the data warehouse, the LTV-IT insisted that the code be run on a server (not a problem). They also pointed out that instead of getting our own server it made a lot more sense to share a server with another department (again, not a problem). The LTV-IT found a server for us – with 275MB available disk space. Now we have a problem. Considering the potential impact of the LTV project 275MB was fairly ridiculous. Fortunately, we were able to design a trimmed-down process that fit in 275MB.

This was where wearing multiple hats on the project became very handy. A design group that was separate from implementation would have made sure that the design was complete enough and robust enough to cover all contingencies, and it would have been a lot larger that 275MB. Because design and implementation were the same we knew exactly where to cut corners.

1 comment:

Anonymous said...

Interesting points on extracting data, For simple stuff i use python to get or simplify data, data extraction can be a time consuming process but for larger projects like documents, files, or the web i tried http://www.extractingdata.com which worked great, they build quick custom screen scrapers, extracting data, and data parsing programs