Migration From AWS to OCI – GovCloud
Hosting on AWS
We setup up the hosting for our commercial Oracle APEX applications on AWS over a decade ago. It has been highly successful and relatively drama free. Our clients now require that we host at least one of our applications within the U.S. government’s FedRamp standards. While approved for FedRamp within AWS, we recognized that we had to rebuild and migrate our home-build Oracle infrastructure to FedRamp.
This gave us an opportunity to undertake our 3rd – or 4th – serious look at Oracle Cloud Infrastructure which we did during the fall of 2023. I have decided to publish this series of articles in December of 2023. The road was not smooth. I think most of our problems were due to differences within GovCloud from standard OCI.
I am very happy to collaborate with Oracle and the GovCloud team to continue working through issues. My colleague and I invested a month in building our needed infrastructure at OCI/GovCloud. We sell/host 2 commercial applications that have been written in Oracle APEX. One application is a time/expense tracking and invoice system used by government consultants. The other application is a grants management system that we offer under a GSA government contract. This application has supported up to 2000 users, 400K PDF documents, and over $5B (USD) in federal grant funds.
We face continuing pressure to migrate our grants management system to FedRamp. We have been approved to host at both OCI/GovCloud and AWS FedRamp.
Please note that several of the solutions presented are awkward and complex. We read, studied, and even opened numerous service request tickets with Oracle during our work. What is presented is what worked on GovCloud during the fall of 2023. We are not presenting all of the failures. Nor are we presenting the cool easy ways that do actually work with standard OCI. We had to experience that some of these just do not work in the GovCloud space.
Contents
GovCloud
Oracle GovCloud is a means of complying with various US federal regulations related to cybersecurity. GovCloud is available for other countries. Please go research. It is fascinating. Because my firm has government contracts and we track financial data and occasionally encounter confidential data, we are eligible to host our data on FedRamp compliant infrastructure.
Status
As of publishing this article on 14DEC2023, we were not (yet) successful with our mission. Two days ago, we got an email from the Oracle technical support team. We had opened a ticket nearly 6 weeks prior. We failed at setting up DKIM as needed to send email from Oracle APEX. I’ll cover the issues in the “Email Article” after it is published. In summary, the tech support team responded by writing (I edited the text for brevity and clarity):
“DKIM … just needs to be set up more manually for now due to limitation in OCI DNS). (Possible good news it that they are preparing a migration that should open up DNSSEC support and allows us to fully automate our DKIM implementation like we already have it in OC1…. It appears that in the future it will be the same process as our Commercial region but for now it does require extra steps that may not be fully documented”.
Oracle SR 3-34805249501
In short, Oracle tech support confirmed our findings. Our guess is that:
- OC1 refers to OCI – Normal/Civlian,
- OC2 refers to GovCloud.
But, we could be wrong given it is not documented.
OCI Improvements
Every year or two, we have paused to take a serious look at OCI. During our most prior assessment, OCI did not yet have all of the features we required. As of 2023, 2 of our 3 objections have been resolved.
Web Application Firewall
While supporting 2000 users in Puerto Rico and managing $5B in federally funded grants, the government of Puerto Rico put a link to our login page on their public website. Within hours, we were being attacked with what felt like and behaved like a distributed denial of service attack (DDOS). While not actually being attacked with intent, the volume of traffic to the login site skyrocketed. The process impacted the ability to users to log in.
With AWS, we tuned the Web Application Firewall (WAF) that was then costing us about $5 per month. We denied all traffic based on a geographical profile. Given we were tasked to manage U.S. grant funds in Puerto Rico, we basically disallowed all other traffic.
During our prior investigation, Oracle had only recently purchased a firm that would help. The cost of the WAF from Oracle approximated $25K/year (versus $60/year from AWS).
Resolved: Oracle now has Web Application Firewall services.
Custom or Vanity URL
We host multiple applications from our database and we use easy-to-remember URLs for these applications. Additionally we host customer service tools, our development work, training applications, and more. Typically, we will have the following
- Application (example.com)
- Customer service for our support team (cs.example.com)
- APEX access (apex.example.com)
- Customer Support (support.example.com)
This was not very feasible with Oracle until 2021 or 2022. Even then the process of custom URL or vanity URL (as they call it) phased in. Early features were oversimplified. Later, the features became more robust.
Resolved: Oracle now supports vanity URL for Oracle APEX applications
In fact, the process is easier, more efficient, and may prove to be cheaper to support than the solution we use with AWS.
SSL Certificates
In order to host custom URL/Vanity URL for Oracle APEX, you need to have wild card certificates for each domain. AWS provides these certificates. You request a certificate then install it on their load balancer. AWS generates revenue for these certificates by an increase in traffic on load balancers and through the monthly lease of load balancers.
Although Oracle ought to be of size, stature, and reputation to be approved as a certificate authority, it appears that Oracle does not (in November 2023) provide SSL certificates.
As such, you must go to the commercial market to buy certificates. As I’ll describe in later posts, the costs can run hundred and hundreds of dollars per year. During our assessment in 2023, the cost of the SSL certificates significantly reduced the suggested savings.
Unresolved:. You must buy SSL certificates on the public market and manage their renewal.
OCI Migration: Go or No Go
As of the drafting of this article (Nov 2023), we have still not fully committed nor fully executed the migration. It is a costly matter that comes with risks. Risks involve stability in our application and impact to our reputation. We have hosted these apps for nearly a decade with zero performance degradation and never suffered an unplanned outage. There were scary nights and difficult times when our systems had to scale rapidly. For a period of time, we were importing 20,000 PDF documents each evening. We had to adjust server and storage often and quickly as we could not forecast the rate of growth. And as mentioned, we needed the WAF to prevent annoying bot traffic from impacting us.
Why not stay with AWS
We have mild reasons for migrating. We own our own Oracle license. Under current rules and pricing structure, this cost is not recoverable after we migrate. We’ll own our license and still pay full price for our Oracle database.
We have no performance issues.
We are very familiar with our home-build infrastructure. We know how to support it. We have pretty maps of the technology. We know the vagaries of various 5xx error messages. This 5xx means Tomcat is failing. This 5xx means that the database is not responding. We know how to perform training drills and practice our response to issues on our infrastructure.
We have 10 years of tuning our monitoring and alert processes. We rarely miss an issue and nearly never get a false positive.
The AWS Costs are predicable and linear.
Recently, I have been annoyed at Amazon. 50% of the products I order from Amazon are crap. So bad, that I often have to throw them out. The bucket I bought last week leaked from all seams. Didn’t someone write a song about a bucket with a hole in it? I sealed all of the seams with silicon and it now keeps water for the chickens. It goes on from there. That purchase appears representative of my (our?) recent interactions with Amazon.
At AWS, our infrastructure appears to be a bit heavier and a bit more costly than it will be at OCI.
Our current AWS Infrastructure
We host our Oracle database on a Linux server (we maintain both and pay annual maintenance to Oracle for the Oracle license). The Oracle database server does not have an public IP address and security rules that require all traffic be sourced from within our virtual cloud network.
We have 2 Apache servers behind our load balancers. These provide custom URL for our applications with proxy and reverse proxy features.
We have 4 TomCat servers behind our Apache servers. We isolate the production traffic from non-production and provide redundancy.
We have a Window server inside of virtual cloud network. This little fellow terminates our VPNs. It also acts as a relay for outbound email traffic and provide other cool tools to our support team.
Why Not go with OCI?
After years of saying “no”, we’re exploring yes. We won’t know the cost difference for several months yet. I will publish them. Two of the three barriers have been removed.
Simpler Infrastructure
With Oracle Autonomous database (ADB), we do not need to host Oracle. We never adopted Amazon’s RDS technology. AWS did not keep the APEX versions current and we were disappointment with the performance. Oracle’s ADB seems superior to our own instance and likely better than AWS’s RDS.
We do not need servers for Tomcat nor Apache. That ought to be a savings. But those savings are likely offset by the added cost of buying wildcard SSL certificates. You can do the math. A wildcard SSL can cost as much as $800/year. A little Apache server runs at $8/month or $72/year. So let’s round that to $100/year to make it simple. 8 little Apache and Tomcat servers equals the cost of 1 SSL certificate. We may spend thousands on SSL certificates.
The process of setting up Vanity URLs with Oracle’s load balancers is easier than using Apache which means likely easier to support and easier to support with a wider range of folks. Adjusting a URL will not require Linux skills.
Our expectations are:
- Simpler Infrastructure – fewer components and easier access to the tools without adding skills such as Linux.
- Improved database performance – Don’t really know how to measure this. It is the promise that the ADB will be faster, more efficient, and easier to manage as compared to our Linux-based server running Oracle SE2 license
- Cost running the same as AWS. If we see an actual savings, then cool and I’ll call it out.
Current Frustrations
During our 3 prior interactions with Oracle, Oracle was aggressive in helping clients migrate to OCI from AWS. They offered consulting services and migration services. They offered discounts for “bring your own license”
In the fall of 2023, we met with Oracle sales yet again. All of that support and discounting is gone. The word was “good luck you are on your own.” Fine. We built at AWS “on our own”. It is a bit disappointing to not find a rich tapestry of well written and current articles about migration. First, the fault lays with Oracle. Second the fault lays with search engines.
Oracle is making it hard and they don’t know it
When you key in searches for OCI migration from AWS or other related phrases, you are first bombarded with pseudo-technical information. The articles have enough technical buzz words and phrases to score highly on search engines. Then the article conclude with “Contact Sales” or even worse a list of consulting services offering to undertake then job for you. In short, there are sales articles being treated as support or technical articles.
When we do find articles that are detailed and technical, they are often obsoleted. Even a great series published by Oracle on creating Vanity URLs observes that the underlying technology changed between part 1 and part 2. And in part 4 of that series of articles, the author informs you do delete half of what you did in the first 3 parts. Oracle has rebranded and renamed their cloud offerings which makes finding current articles difficult. It is just too easy to find information that is for the wrong version or articles that have been obsoleted as OCI improves.
GovCloud setup, menus, and processes are totally different and documentation is un-findable (by us, Google, and a pair of Ducks-going).
GovCloud Articles
In conjunction with the above statements, finding information about OCI GovCloud was very difficult. The processes within GovCloud are simply different which means to discover this, you need to isolate the articles on search terms, taxonomy, and topics. DKIM setup or DNS setup for OCI is entirely different within GovCloud. No articles identified these differences.
Search Engines make it worse
Search engines prioritize results based on the domain owner’s name. If you search for “Oracle OCI”, your first pages of results come from oracle.com . The articles of the people who have been-there-done-that get buried deep behind Oracle’s sales and Oracle authored articles. That’s life on the modern internet.
I am at least 2 years later than others in tackling this. And yet, I struggle to find the articles I am accustom to finding by well-known Oracle bloggers. Frankly, I need the been-there-done-that article because they accommodate the honest variations and work-around that I need.
Plans, Challenges, and Steps
The checklist for success includes:
2 – Migrate a database pump file from AWS to OCI
3 – Install a database pump file on OCI into specific schemas
4 – Have functioning applications (kind of obvious, huh?)
5- Be able to send email from Oracle APEX to users
I’ll write and publish an article for each step including tricks, failures, and successes.
