“All models are wrong, but some are useful” (George Box, 1978).
Now, although I agree with the sentiment of the statement, we need to make a clear distinction between the design of a business model and model outputs. Over the past 15 years, I have built many business models, predominantly in a healthcare setting and I’d like to think that these models have been technically accurate, representing the data and assumptions upon which outputs have been derived. Whether the outputs and recommendations derived from the model become reality is a separate consideration.
Irrespective of whether the outputs are perfectly accurate, the key to successful modelling is the ability to point people in the right direction – to better understand the potential impact of certain scenarios and to help them to seek answer to questions which will help them make more informed decisions.
"the key to successful modelling is the ability to point people in the right direction"
As someone who has been producing data models within healthcare for many years, I thought it time to share some of the lessons I have learned and continue to learn and distill some of this learning in to a series of articles under the banner ‘The Art of Healthcare Modelling’.
There are so many topics to cover, but ‘scoping the work’ seems like a great place to start. Scoping is predominantly about the what and less about the how. That said, as you’re developing the ‘what’, you will undoubtedly be thinking about the mechanics of delivery.
It might seem obvious, but modelling projects which are poorly scoped are more likely to hit problems before completion. A lack of scope can lead to drift and misalignment between parties. A well scoped project makes it easier to manage customer expectations, whilst also providing customers with reassurance and the ability to hold you to account.
Understanding the questions the customer is seeking to answer, (the what), and how this information will be used is vital, (for me it makes no difference whether the customer is internal or external to your business). If possible, doing this directly with the customer is hugely beneficial, else there is a risk of misinterpretation. If you, (or a team you’re part of), are going to build a model, it’s much easier to do so when you have the necessary context, rather than relying on the heresay of colleagues who may predetermine what they think is important. It can also help to build trust and confidence in the relationship with the customer, dispelling the myth that modelling is a black box exercise.
Although face to face meetings are not always possible, in the era of mobile technology and communications, you don’t necessarily need to be sat in the same room as your customer. If you’re not comfortable or don’t feel confident enough to lead such a process, use a trusted colleague, but at the very least, try and join the conversation to get first-hand insights.
Experience has taught me that you often need to probe deeper than what’s front of mind for the customer. Consider using a coaching approach to let customers arrive at the answers themselves, (co-design and collaboration is another topic to be covered later date). Don’t be afraid to ask lots of questions as this may help to unravel additional needs and requirements.
Example: In the course of my career, I have seen many tenders which simply state the need to understand how many inpatient beds will be required in the future. A simple place to start is to ask why the customer needs to know this? Is this for a business case, is it to inform a one-off report for a senior executive? Is it to support operational performance?
Understanding the intended purpose of use really helps set the context and helps determine what questions to ask next. Other considerations and probing questions could include:
- How far in the future do we need to model?
- Does the model need to differentiate at specialty/diagnosis level?
- Does the model need to differentiate by different types of patients?
- Is the age and gender of patients important? If demographic growth is a key factor, then the likely answer is yes given the demands the older population place on healthcare services.
- What initiatives might need to be considered such as moving activity from one hospital to another or repatriating activity from other providers.
- Do you need to model changes to the average length of stay, or do you want to model changes to the distribution of length of stay?
- Are there dependencies such as theatres and diagnostics which we need to consider?
- Is seasonality a factor – do we need to understand outputs by day/week/month/year?
Answering these and other questions will really help you understand and shape the model – helping develop a picture of the assumptions are that need to be deployed within the model.
Although not a comprehensive list, by asking such questions, what started as a simple stated need is now more contextualised, with far more information wrapped around it, which really helps to scope the scale of the work required. Such scoping questions will also ensure that the model you build will be fit for purpose. As I would say, there’s no need to build a palace when all you needed was a shed!
"there’s no need to build a palace when all you needed was a shed!"
By determining what the model needs to do, you’re also starting to build a list of ‘out of scope’ statements for what it will not do. This is extremely important in terms of setting and agreeing expectations.
It’s also important to understand who will be using the model and how will they need to access it? What skills do they have? What technology and systems do they use? A very recent experience reminded me of the need to understand the hardware and software that the customer has. If you’re developing a complex Excel model on a high specification computer, consider how the customer will access this, particularly in context of public sector organisations where computer hardware is often of a lower specification due to budgetary constraints. As part of the scoping conversation, it would be helpful to understand this as it may affect the way the model is developed.
In terms of managing expectations, it’s really important to understand the time-frame and budget that the customer is anticipating for the modelling project. If the modelling is part of a larger piece of work, it’s still worth trying to understand expectations around discrete elements of any modelling work. Based on what you’ve scoped, you’ll start to get a feel for whether an indicative budget is realistic. If it’s not, you’ll need to be honest – don’t head down a path which is doomed to failure or disappointment. If the customer is constrained by a specific time-frame and/or budget, consider what you could scope which would fit within this - relationships and outcomes will always be better when they account for mutual interests.
Scoping doesn’t stop there, as you still need to formally agree what you’re going to do with the customer. Depending on the scale of the modelling, you might wish to write up the requirements in the form of a project charter as this gives everyone involved clarity on scope and can be used to both set and manage expectations.
- Understand what questions the model needs to answer.
- Ask lots of questions about how the model will be used and by whom.
- Ensure the model is fit for purpose, don’t add assumptions that won’t be used.
- Make sure the model will work on the customer's computer system.
- Manage expectations and be clear what is both in and out of scope.
- Co-design will increase the chances that the model is successfully adopted.
- Draw up a project charter and get formal agreement on what you’re going to do.
It’s probably fair to say that there isn’t a one size fits all solution to scoping a model and we’ve really only scratched the surface, but hopefully this article has touched on some key points which you might wish to consider next time you need to directly or indirectly support the development of a data model.