Recently I had a client decide to implement a major architecture change to their production environment. Of course, this change was going to be made to their production environment, during financial close, and on a weekend. As managers of EPM Systems surely know, there is no ‘good time’ to push major system changes. In this case, the change was moving all the EPM components to a new database (DB) server.
The client in this instance had 13 EPM servers in their production environment. The DB host for most of their instances (HFM excepted) was migrating from one host to a new host. None of the schema names or passwords were changed, only the instance/host name in the TNS string. With HFM out of the picture, this meant consideration had to be given to reconfiguring the following EPM Products:
- Essbase Administration Services (EAS)
- Essbase Provider Services
- Financial Data Close Manager Enterprise Edition (FDMEE)
- Oracle Data Integrator (ODI)
- Hyperion Foundation (Shared Services and Workspace)
- Oracle Business Intelligence Enterprise Edition (OBIEE)
- Reporting & Analysis
- Financial Reporting
- Financial Close Management (FCM)
- Account Reconciliation Management (ARM)
- Oracle Service Oriented Architecture (SOA)
Plan & Actions:
There are a couple of ways to approach a change like this, you can simply re-configure all the products using the built-in EPM System Configuration tool, but that has a couple of drawbacks. The one that was most critical here was that a redeploy of the Java application would reset all the environmental tuning. Instead, the following approach was used:
- Run the EPM configuration tool to update the Foundation (Shared Services pointer). This needs to be done on every server, so each server knows where the EPM System Registry is stored.
- Run the configuration tool a second time, for each individual product, to make sure they know where their new DB host is at.
- Login into WebLogic Administration console and change the DB Connection Pool information for each product (so the Java components know where their DB connection is)
- On the backed of the system, in the file structure, delete the /tmp and /cache folders for each product using a JVM. This will force a redeploy, but not impact the Windows service tunings set in the registry. It’s also recommend to clear the Java /logs folder at this point to ensure a clear view of what is happening on restart.
- ODI requires manual changes to files like opmn.xml and the ODI Agents in the ODI Console.
Fun with SOA:
Oracle SOA was the big outlier here. After some team debate, it was thought that we could simply run the WebLogic Domain Administration utility and ‘extend’ the existing EPM Domain which included SOA, and reset the credentials and redeploy. On the day of the change, it was discovered that these features are all ‘greyed-out’ and thus unavailable. Instead, it was decided to change all of the pointers to the new DB server for SOA in the individual registry files found in the /jdbc folder beneath the WebLogic domain. A restart of SOA proved that this worked and there were no unexpected errors. I will note that this only worked because the instance/host name for the DB was all that changed. Had the client decided on a password change (which is encrypted) we would have had more difficulty here.
At the end of the day, we had a successful migration using the high-level plan outlined above. However, it is worth noting that several products (Foundation and RA Framework) did not play nice when first being restarted. This move required a high-degree of technical involvement, the ability to understand how all the IT stack layers interact and some deep log diving to finally resolve all the issues. While a success, it is not something I would recommend a client undertake lightly, or without both a solid backup plan and professional services assistance.
For more information or help re-hosting your own EPM database, you can contact me at firstname.lastname@example.org.