In our Common Misconceptions in Optimization post, we mentioned a couple strategies to try if you’ve tried optimization and it didn’t work out, but we wanted to go into more detail, as it seems to be a common issue.
We’ve all worked with contractors or consultants that didn’t deliver what we expected, and bad experiences are made worse when the final product itself doesn’t deliver.
As we mentioned in the previous post, we generally believe disappointment in working with optimization companies comes down to two factors:
Optimization technique limitations (‘power’): the technique that was being used wasn’t powerful enough to handle your problem in a reasonable amount of time.
Flexibility: the tool/provider/solver you used wasn’t flexible enough to generate results that were useful to you (and this is often a result of the mathematical technique used as well).
So, in general, there are two things to think about when evaluating a poor experience with a provider:
Are most of the issues related to the provider (consultant issues), or the underlying technology?
How do we identify and solve these issues?
A General Framework for Identifying Problems
The first step in moving on from any form of contract work is to identify what went wrong, so it can be avoided in future. Note we use the terms contractor/provider/consultant interchangeably throughout.
Usually there are a few steps to this:
Identify what you believe to be the root causes: ideally, have separate debrief meetings with the team members involved, and discuss why they think the project went wrong. Make sure to discuss both the things they believe went wrong on the consultant side, and what things they think went wrong on your side. Both are important.
Expand this study outside your company: ask around your industry and try to engage other companies who have had negative experiences with the same firm; don’t colour their opinions, just try and learn the same things as above: what went wrong, what do they believe caused it, and what do they think they could have done to improve the situation?
Try and find some success cases: now it’s time to flip things around - what companies had success with this firm? Why do they believe this happened?
Synthesize: At the end of this process, try and combine this information to identify the root causes of failure that are specific to your company, and specific to their company.
Use this framework to identify the general issues between you and the provider.
Some common root problem causes when working with consultants:
Mismatch of project deliverable expectations from the beginning.
Too large/ambitious a project for too small a time frame.
Related: poor estimation of time/expense for a given deliverable.
Mismatch in expected communication.
Usually caused by poor outline of expected communication (ie. winging it).
Identifying Problems with Optimization Technology
We mentioned what we see as the two most common causes of optimization failure earlier: flexibility and power.
Ultimately, you’re going to need an optimization/operations research expert (see #7) to truly verify the technological limitations, but generally it will come down to two things:
Problems with usability of results: this is things like the inability of a product/provider to get to results that are directly usable, a large number of invalid routes/schedules, limitations on the types of rules that can be added, etc.
These issues indicate a lack of flexibility from the optimization technique or provider.
Problems with speed: this can be a very long runtime to solve a given problem, runtime issues when a large number of rules/variables are implemented, a large amount of manual intervention required to get a solution, etc.
These issues indicate that the power of the optimization technique is lacking.
There is one other potential issue that warrants mentioning, and that is stability. An optimization product/algorithm must be stable to be useful.
Indications that a product may not be stable include:
A large number of invalid results in the solution.
An inability of the provider to correct invalid results.
Persistent invalid results or inconsistencies in the solution.
A large amount of manual intervention required to obtain a solution.
Hopefully these frameworks give you some insight into identifying issues with both a particular provider, and with the provider’s optimization product.
Often, the most difficult part about evaluating why things go wrong is because there is very rarely only one issue. Usually there are several, and often they affect each other, making it difficult to distinguish the root cause.
Our best advice on this front is to try and approach your evaluation objectively, and proceed step-by-step. Often chatting with others who have used the same provider will give you the final key to the puzzle, highlighting some of the issues you’ve identified as consistent problems.
Should You Try Again?
At this point you’ll have to make the tough decision about whether to try again with another optimization provider.
We firmly believe that implementing proper optimization in your company can make a massive difference to your operations, and ultimately your profitability and growth, but we’re also biased.
At the very least, we’d suggest you follow the steps we outlined in How to Select an Optimization Company to at least look at other options, and you can take your previous experience into the process.
We would still recommend you complete all steps in the process; however, when you get to the step of qualifying firms via a call or demo, you should have some much more pointed questions to ask about how they work, their products, etc.
We would also recommend, that like hiring, you should ask behavioral questions of the firm, aimed to elicit insight into the problem areas you found previously. Example questions might be asking about how they typically communicate, how they deal with problems/bugs, etc.
This should help ensure that you avoid your previous mistakes, and give you the highest possible chance of success the second time around.
Any more tips for dealing with consultants or software providers? Let us know in the comments!