Search code examples
amazon-web-servicesoracle-apexaws-elb

Oracle APEX wwv_flow.accept call yielding 403 forbidden when using AWS ELB

We have an Oracle APEX application hosted on two EC2 instances, with an AWS application load balancer, with "Sticky Sessions" enabled. The APEX application works fine 99% of the time, but when a process attempts to upload a file, or a couple other places which invoke the procedure wwv_flow.accept, we get a 403 Forbidden message.

We have an Apache 2 httpd server, ORDS running standalone, and APEX. Latest versions on all; version details at the end.

There is not any log messages for the 403 in the Apache logs, nor in the ORDS activity log. We have done a system trace of Apache and that yielded nothing useful.

We have turned off the load balancers and the issue goes away. We have replicated the issue in a separate AWS environment.

In Google DevTools, we see this; note the "Remote Address" of 3.208.141.138 port 443, and the "Server" of "awselb/2.0"

Dev Tools Network

Doing an ICANN lookup on 3.208.141.138 yields a Registrant Name: Amazon Data Services NoVa

I assume the "Server" value of "awselb/2.0" means Amazon Web Services Elastic Load Balancer version 2.0.

So it appears that the load balancer is returning the 403 error, and "403 Forbidden" web page.

We opened a ticket with Oracle, and they couldn't find any issues with our APEX nor ORDS setup.

We captured an HTTP Archive (HAR) file for Oracle support, and here is the request and reply from that capture:

Request

        "request": {
          "method": "POST",
          "url": "https://c333.example.net/ords/wwv_flow.accept?p_context=app-builder/import/14656898533235",
          "httpVersion": "http/2.0",
          "headers": [
            {
              "name": ":authority",
              "value": "c333.example.net"
            },
            {
              "name": ":method",
              "value": "POST"
            },
...
           ]

Response

        "response": {
          "status": 403,
          "statusText": "",
          "httpVersion": "http/2.0",
          "headers": [
            {
              "name": "content-length",
              "value": "520"
            },
            {
              "name": "content-type",
              "value": "text/html"
            },
            {
              "name": "date",
              "value": "Mon, 16 Sep 2024 22:06:38 GMT"
            },
            {
              "name": "server",
              "value": "awselb/2.0"
            }
          ],

Software Versions

Operating System: Oracle Linux Server release 8.10
Apache:           Apache/2.4.37 (Oracle Linux Server)
Oracle database:  Client-managed Version 21.14.0.0.0
Oracle ORDS:      24.2.3.r2011847
Oracle APEX:      24.1.4   API_COMPATIBILITY:  2024.05.31 Patch:  APPLIED
Solution

  • Problem solved; We had to modify our rules for our Web Application Firewall (WAF) relating to restrictions on the size of certain types of requests. The file uploads were larger, so an increase of sizes solved the problem. The Amazon WAF was the source of the 403 Forbidden page.