30 Years in Tech: My First ‘Data Warehouse’
This year marks three decades as a professional in the field of data & analytics, which seems a valid reason to reflect on all the silliness I’ve seen during these years and whether things have improved since the start of my career (spoiler: not necessarily).
Back in 1991 I was a fourth year information science student with too much time on his hands and eager to do some real work. A bit of extra cash didn’t hurt either, so I signed up at Integrand (student internship mediator) and soon got a call for an interview at an SMB near by. The customer turned out to be an ambitious holding company looking for a better way to consolidate and report their financial information from the different companies, each one with its own finance application. The ‘system’ they had at that time consisted of a bunch of unrelated spreadsheets and sytem export files which were sent in on floppy drives. Then, it took them another week or so of mostly manual work each month to finalize their monthly management reports. In total the process took 2 to 2.5 weeks each month, which was far from ideal. So my assignment was pretty straightforward: figure out how to speed up and improve this process. The goal? Have their numbers ready within one week of monthly closing.
Did I succeed? Well, let me start with bragging about the end result: I got the whole thing down to within a day (in the rare occasion where everyone was on time delivering their data), but regularly it took about two days. Needless to say they were very happy, and since I love happy customers (I still do btw), I was happy as well.
The project wasn’t without its challenges of course. It was 1991. There was no network between the various offices, so data exchange was very physical: storing data on a floppy drive and send it in, or even: bring it in. All companies had their own finance system, their own KPI’s and different markets in which they operated. And yet they needed insights in how the company as a whole was performing. Fortunately the guy who hired me was an experienced controller with some IT skills, and I had a background in data and IT and had basic finance knowledge. Together we made a pretty good team. Which brings me to the first ‘best practices’ I’d like to share: make sure there’s domain expertise in your team, and (even more important), be aligned with the business and business goals.
What we ultimately defined was a set of over 60 data points that were common across all companies, and which covered everything required for the reports and KPI’s they needed. To collect this data I developed a data entry application where every company could simply open the current month and start entering the required numbers. For the storage and consolidation of this data I developed a multi dimensional database, which served as the source for the KPI, trend, and detail reports that were also included in the same application. And all of this without any knowledge of data warehousing or business intelligence, terms I learned about only years later. We just figured things out along the way and did what seemed to make sense. Apparently, we seemed to have figured out the right things to make the project a success.
But wait. Multi dimensional database, KPI reporting, in 1991? Yep, all of this was made possible by the latest version of Lotus 1–2–3: 3.1! Version 3 was the first spreadsheet application with multiple worksheets (‘3D spreadsheets’ which was exactly what we needed), and version 3.1 added WYSIWYG capabilities making the whole thing a lot more user friendly. Add in the basic database operations, a powerful macro language and integrated charting/graphing and you had a pretty powerful workbench for the task at hand. The solution and workflow was pretty straightforward: each subsidiary had its own floppy with the data entry application. Every month everyone entered the required numbers and sent in the results, after which the data was imported in the central ‘database’. All that was left then was pushing a button to start the macros that did the rest of the magic.
This brings me to the final ‘best practice’ or insight of this post: technology is a great enabler (and can even be essential), but real success can only be achieved by proper design, process, and business alignment. All of these have very little to do with the latest IT gizmos.
