Big Data, Huge Hype: A Realistic Look At How To Prepare Your Organization

I have finally given up trying to refrain from saying “Big Data”, but I still hate the term. Big Data does not yield any information if the data is worth its weight in the hard disk drives it is stored on. It is similar to saying “Big Numbers” for a large corporation’s income statement. You can have a billion dollars worth of revenue and yet never make a dime of profit — is that good revenue or not? While Big Data is all the rage now, it is teed up to be the next technology that executives complain about when referring to IT investments.

Before you jump into the Big Data craze, or if you have started to jump in, take some time to understand what you are getting your organization into.

Be realistic about the state of big data

Big Data technology, consulting, groups and vendor projects are being marketed hourly to us, but the reality is we are just now starting to understand the potential value of mashing all of our data together. This is new territory and there is not a maturity model or direct path to follow. I see this as looking for low-hanging fruit during a moonless night. It is really dark out, and while we may trip over some fruit, we barely know where the tree is, and it is almost sheer luck that we were able to derive that benefit from it.

Certainly some organizations are jumping on the bandwagon, and organizations that are “data mature” have found value in the investment. The key to finding value, in my opinion, is knowing if your data is mature enough to bring together.

For instance, in health care, how robust is your enterprise master patient index, and how sophisticated are your admitting clerks? What is your data duplication rate, and how quickly are you doing proper combines? If you have solid process that is being followed, you have worked out the kinks related to your master patient index, and you are keeping up on your patient combines, you might have a solid case to begin a big data project. However, if you are struggling to get your teams to not create a new patient whenever they present at the admitting desk, or if you are not managing your potential duplicates properly, you probably have no business diving into Big Data projects.

What business problem are you trying to solve?

The next time you’re at a conference or business function and someone says they are focused on Big Data, ask them what business problem they are trying to solve, and when their expected payback will be. Many times you can hear the crickets chirping in the silence and glazed-over look. Big Data projects should be treated like any other business investment. There is a cost and there should be an expected return. Will the return be positive? Maybe not upfront, but you still have to do the analysis.

I’ve started to get requests from customers where they lead with the question “Should we invest in Big Data? Because I need to get information on XYZ.” The same marketing teams that are barraging CIOs with Big Data needs are letting our customers know that Big Data is the next thing they should be pushing for. For those requests, I was able to leverage a tool that we already have to solve their request. Now could we do it with Big Data? Sure, but it is not the answer for everything.

Big Data should not be the reason you do a project. It would be like saying “Laptop” and hoping that someone would fund your next replacement machine. Work within your organization to develop why you might need to invest in Big Data and what the returns might be. This leads to some really interesting non-technical thoughts about Big Data.

As an example, with population management, we might now be able to tell which patients are at a higher risk, related to a specific disease; however now that you have that data, what are we going to do with it? Will the patient still be compliant if they have not been compliant to date? Who is going to call them and what is the payment mechanism going to be when we ask them to show up for preventative care? All of those questions should be answered alongside a Big Data investment analysis that has little to do with the technical aspects of the project.

Be realistic on the challenges you’ll face

Big Data projects are not going to be easy. I have already referenced the challenges with patient matching, but we also need to consider how the data is going to be used and if it is actionable and correct. Many times we have looked at bringing large databases together and have found that the data cleaning, nomenclature differences, and disparity between data definitions has created massive challenges. Two systems can be built separately and function perfectly as standalones, but when you talk about smashing data together and making it meaningful, the build decisions that were made independently increase the amount of complexity related to making that data meaningful.

You should also realize that this is a relatively new endeavor for many of us. There will be capital investment involved, and because the technology is improving, the amount of reinvestment in the short term can be high. Add to this the cost of either consulting assistance to augment the knowledge gap, or the training required to properly manage and support the needs of the organization, and it can be challenging to keep on top of the big data investment. This risk also makes big data ripe to be the next complaint about technology costing millions but never returning any substantive savings to the organization.

Start small

You cannot and should not ignore big data. The technology does have potential and you do not want to be left behind. In my opinion, you should start preparing by doing the following:

1. Continue to stay on top of what’s going on. Find larger peer organizations that are utilizing Big Data to solve interesting questions. Figure out how they are doing it and determine if you have the same challenges and start building a business case to see if the time is right to start your own project. Find a company outside of your industry and ask them how they are using Big Data to solve problems, or how they are thinking of leveraging Big Data.
2. For every new system, consider how it should be built to have the data combined with existing systems in the future. Nomenclature similarity, build consistency, and openness are all things that should be considered during a new software acquisition. There will be a day in the future when most of our databases will be combined — do not make future work harder because of short-sighted decisions today.
3. Clean up your existing data. Because our data will be smashed together in the future, it is never too late to start enforcing policies and cleaning up existing data sets to make the future investment more successful. If your admitting department creates too many duplicates, then roll out additional training and education around why that is happening.
4. Meet with vendors now about their intentions related to Big Data projects — what are they doing, what products are they leveraging, and what success have they had. You cannot learn too much and it is best to get as much learning for free before committing to a platform or tool. If you can, partner with them on a small pilot project to get your feet wet without investing millions of dollars.