An on-premise ERP upgrade horror story (and a permanent solution!)

Project Background

Ok, well not really. As on-premise ERP upgrades go, this one went fairly well. The company is a mid-sized enterprise with a moderate level of system customization. Compared to other upgrades (and I have done a lot of them), this one was in the middle in terms of complexity and risk.

In addition to the ERP software upgrade, the company had to upgrade their database to the latest version, but they were assured by the ERP software vendor that this was not a big deal and that the application software would work flawlessly with the new database version. Since the database vendor and the ERP software vendor were one in the same, what could possibly go wrong? The upgrade project lasted approximately six months. The project was well staffed with functional and technical experts and an engaged user base. We had an active steering committee comprised of executive stakeholders from across the company. The project ran like an upgrade project should as we hit key milestones for development, testing and training. We stress tested the application with production data and compared the output against actual results in the old release. Overall, the project was going well.

Four Uh-Ohs

So why did the project sponsor, who was new to the world of on-premise ERP, say “I never want to do another ERP upgrade project ever again”. There were four main reasons:

1. Mysterious Performance Issues

We tested the application at production level volumes for all major interfaces. In the test environment, processes ran at or better than the production run times. We were pleased. After go live, key processes that ran in minutes on the old version were running for hours (and hours and hours). What happened? It worked in test. It worked previously, and we hadn’t changed any code. We called our talented Database Administrator (DBA) in to help resolve the issue. Then we waited. The DBA worked closely with the ERP software/database vendor, as well as the network and server administration teams. He reviewed logs and cases. He was stuck and so were we. We had to close the month, and we were stymied because somewhere deep in our technology stack something was not configured correctly. After nine days of discovery, someone suggested that we try and switch the Dirty Data Trust Factor Switch (ok, that was not the real name, but it was something equally ridiculous) from on to off. And ta-da – it worked! Why it worked, we did not know, but we took it as a win and moved on. The project sponsor asked naively, but insightfully, “Why would we ever need to know what this switch does?”

Good question – it should have all just worked, right?

2. The Long Weekend of Angst and Hope

The project ran for ten months. We all worked very hard to understand the new features and how they applied to our business. We re-engineered the chart of accounts, cleaned up our vendor master and re-designed security. We streamlined key processes. We trained our users, and they tested the heck out of it. Then came the go live weekend. We were all now reliant on our trusty upgrade administrator. He is a talented fellow and worked hard throughout the project, but now we handed off the upgrade to him, entrusting him with 120 hours of steps, tasks and scripts that needed to be completed. We set up a war room and had everyone on call. Then we waited. The 120 hour plan was going well until about hour 68. That is when the train went off the track. The process steps that ran in minutes were now running in hours (sound familiar?). Our administrator tried to explain what was happening, but there was not much we could do, aside from wait and watch the processes creep along. As the processes ran longer, we started to get paranoid – the process finished, but did it work? We started QA-ing the scripts, and it threw out tons of false positive errors as the rest of the scripts had not yet been run. Did we have a problem or not? The upgrade scripts are long and complex, but what else could we do at this point? We decided to move on and run the rest of the scripts. We ended up finishing the upgrade at hour 114, eating into every bit of buffer that was built into the plan, but still going live on Monday morning. Our heroic upgrade admin made it through the weekend but was visibly exhausted. The project sponsor quietly noted, “Wow. We would have been in serious trouble without that guy.”

Good point, why does it take so much effort, specific expertise and faith to upgrade your financial applications?

3. Lots of Work and No Real Value Delivered

During the upgrade, our sponsor noted that so many of the company’s resources were working on the project. Some of the resources were dedicated full time. Others had to work additional hours or defer other work to be trained in and test every single part of the application. We had hundreds of issues related to the upgrade scripts, the customizations, the new features, the interfaces and of course the performance. This was a lot of work. This was a major allocation of time and effort at the cost of other possible activities. About half way through the upgrade project, our sponsor asked about the value we were gaining from the upgrade. Although there were some noticeable benefits in several process areas, like collections, most processes would remain pretty much the same. This legacy on-premise ERP had been around for decades, and the upgrades became less and less impactful in terms of delivering new value. The project sponsor once again asked an innocent, but spot on question that all CFOs and CIOs should be asking, “So why are we doing this project?”

The answer was simple, though unsatisfying – because we have to keep up with the ERP software vendor’s release to stay on support.

4. Is this the latest version of the software?

As mentioned previously, the application spanned not only finance, but hundreds of other casual users of the application. We were doing one of our training sessions for these users, and it was going well. We covered the new process and what was changing. We did live demonstrations. We fielded many questions and everyone seemed content. At the end of the session, one user raised his hand. He thanked us for the training and apologized for the question he was about to ask – “Is this the latest version of the software?” Everyone chuckled, and I assured him it was. He accepted the answer and moved on. But why did he ask the question in the first place? He was asking the question as a consumer not a transaction processor. He wanted the application to look, feel and interact with him like other applications do in his daily life. He wanted it to be easy to use and accessible not only from his desktop but on his mobile device. The on-premise ERP software vendors seldom deliver ground breaking innovation in their releases. They are typically just adding a few bells and whistles to millions of lines of code that have been developed over the past three decades.

The reality of on-premise old ERP is that they have a hard time innovating. Their code is complex and built on an “upgrade” approach that just adds fixes on top of fixes on top of fixes. They can’t move fast enough to keep up with the pace of technology.

And now the rest of the story . . .

After upgrading the financial system, our client had a new challenge. The same ERP software vendor informed them that they will need to start thinking about upgrading their other platforms. Our sponsor wisely asked, “Is there another way?” Thankfully, there is. Over the past five years, the popularity of cloud applications for finance has exploded. A 2013 Gartner study (Adoption of Cloud ERP, 2013-2023) shows that 47% of companies plan to move their core ERP systems into the cloud by 2018. CFOs are moving to the cloud for many reasons; however, some of the main benefits include (and address our sponsor’s uh-ohs!):

  • With cloud applications, you subscribe to an application and focus your efforts and energy on your business, not the technology stack. This means you never need to know about the Dirty Data Switch!
  • The cloud application provider takes care of the technical portion of the upgrades, releasing them two to four times a year. The upgrades take place over a few hours and allow you peace of mind that experts are taking care of the upgrades not only for you, but for all of their users at once!
  • There is some work with cloud upgrades, but at a fraction of the cost and effort. Your focus is on understanding and deploying the new features, completing regression testing, and making sure your interfaces and reports still work (they should!). You measure the efforts in days and hours, not months.
  • Innovation is constant. The cloud applications are based on new technologies that enable innovation. They are often built on open standards vs. proprietary languages and third party add-ons. They are built for how we work today (mobile), not how our parent’s worked!
9 things CFOs love about the cloud
Changing Financial Systems for CFOs
Related Posts
Equal Pay for Equal Work Compliance in Workday
Equal Pay for Equal Work Compliance in Workday
Deploying Workday: Too Many Applications Result in Too Much Risk and Inefficiency
Deploying Workday: Too Many Applications Result in Too Much Risk and Inefficiency
Easy Integrations: Prebuilt Connectors Speed Deployment Times
Easy Integrations: Prebuilt Connectors Speed Deployment Times