CVE-2022-31813: Forwarding addresses is hard

Written by Gaetan Ferry - 26/07/2022 - in Exploit , Pentest - Download
A few weeks ago, version 2.4.54 of Apache HTTPD server was released. It includes a fix for CVE-2022-31813, a vulnerability we identified in mod_proxy that could affect unsuspecting applications served by an Apache reverse proxy.
Let's see why it is rated as low in the software changelog and why it still matters.
TL;DR: when in doubt, patch!

Underlying concepts

If you are already familiar with HTTP proxy chains, and particularly the transmission of requests' information from a client to an app, you may skip to So, what is wrong with mod_proxy ?. If you only want the digest of the story, skip to Wrapping up.

The good application, the bad IP address and the ugly proxy

The time when all web applications were exposed directly on the internet is long gone. We now hardly see web infrastructures that do not contain a reverse proxy, load balancer, or both, that fronts the application(s). The reasons for that are numerous, performance, costs, IPv4 exhaustion, etc. This evolution in the standard architecture brought a new set of issues and technical challenges.

One of these problematics is source IP management. When using a reverse proxy (or other fronting component), HTTP requests sent to the application containers are issued from the IP address of this equipment and not the real client one. This is a problem when applications need this information, which is often the case, be it for logging or filtering purposes.

To solve this challenge, reverse proxies started injecting information about the original HTTP request in headers that are forwarded to the application. Those include entries for the client's IP address, the originally requested host, the protocol, etc. Up until (not that) recently, no standard field existed in the HTTP specification to store this information. Therefore, non-standard headers have been created:

  • X-Forwarded-{Host,For,Proto}
  • X-Real-IP
  • Client-IP

X-Forwarded-For, X-Forwarded-Host and X-Forwarded-Proto have then become the de facto standard. This still holds true, despite the publication of RFC 7239, in 2014, which standardized a Forwarded header field.

The operating of this headers is pretty straightforward. The values in them represent a piece of information relating to the original client's request:

  • X-Forwarded-Proto: contains the original request protocol (i.e. HTTP/HTTPS).
  • X-Forwarded-Host: contains the original value of the Host header as received by the proxy.
  • X-Forwarded-For: contains the original client's IP address, along with the IP addresses of all the previous intermediate proxies.

In essence, that's all for the forwarded headers.

Hop hop hop-by-hop

Hop-by-hop headers is an HTTP mechanism that allows instructing proxies about which headers should be forwarded along a request (end-to-end) and which ones should be immediately treated and dropped (hop-by-hop).

There are a few headers that are always considered as hop-by-hop. They are:

  • Connection
  • Keep-Alive
  • Proxy-Authenticate
  • Proxy-Authorization
  • TE
  • Trailers
  • Transfer-Encoding
  • Upgrade

While all other headers are considered end-to-end, it is possible to add some of them to the preceding set, by listing them in the Connection header, right after the classical Close or Keep-Alive keyword. Doing so, all listed headers should be treated and dropped by the first proxy that receives them.

This mechanism exposes an interesting attack surface, as explained by Nathan Davison in his 2019 article [HBH].

So, what is wrong with mod_proxy ?

As all reverse proxies, Apache HTTPD with mod_proxy tries its best to follow both the real and defacto standards. This means the proxy component will add the X-Forwarded headers, as stated in the documentation.

When acting in a reverse-proxy mode (using the ProxyPass directive, for example), mod_proxy_http adds several request headers in order to pass information to the destination server. These headers are:

  • X-Forwarded-For: The IP address of the client.
  • X-Forwarded-Host: The original host requested by the client in the Host HTTP request header.
  • X-Forwarded-Server: The hostname of the proxy server.

In the meantime, the proxy also respects the hop-by-hop headers indication and removes them from the forwarded request.

The problem is, if the X-Forwarded headers are listed as hop-by-hop, Apache fails at forwarding them to the upstream infrastructure. In a normal situation, when receiving a legitimate request, for example from this curl call:

$ curl myhost.local/test

an application behind an Apache mod_proxy reverse proxy receives the following HTTP request:

GET / HTTP/1.1
Host: localhost:5000
User-Agent: curl/7.74.0
Accept: */*
X-Forwarded-For: 192.168.42.42
X-Forwarded-Host: myhost.local
X-Forwarded-Server: 127.0.1.1
Connection: Keep-Alive

But if the same request is sent, including the X-Forwarded headers in the hop-by-hop list:

$ curl -H "Connection: close, X-Forwarded-For, X-Forwarded-Host, X-Forwarded-Server" myhost.local/test

the received request is different:

GET / HTTP/1.1
Host: localhost:5000
User-Agent: curl/7.74.0
Accept: */*
Connection: Keep-Alive

What is interesting here, is that the received request is absolutely no different from one that would have been sent from the loopback interface directly on the application server.

The cart before the horse

This weird behavior can be quickly tracked down in the source code of the mod_proxy HTTPD module. Everything interesting is packaged in the proxy_util.c file (modules/proxy/proxy_util.c).

The function responsible for handling hop-by-hop headers is ap_proxy_clear_connection. It is even nicely indicated in the code documentation.

The code snippet responsible for adding the X-Forwarded headers is located in the ap_proxy_create_hdrbrgd function. Once again, a nice code doc indicates the feature.

What is important is that, while the X-Forwarded addition code is called at line 4050 (or so), the ap_proxy_clear_connection function is called AFTER that, around line 4083.

You got it now, mod_proxy, as bundled Apache HTTPD version 2.4.53, first fills the X-Forwarded headers before removing them right away. This is a fail.

This issue was introduced in version 2.2.1, more precisely in revision 377053 on the 1st of April 2006 (a bad taste April fool). Which makes this bug a 16 years old one.

How bad is that?

Well, you are probably not going to compromise an Apache server with that issue. The fact is, this issue falls in the "application impacting" category. Its consequences will depend on the application and infrastructure setup.

There are (nearly) no generic, evergreen, attack scenario. Instead of that, multiple use cases can be found, depending on the context.

WEN ETA attack scenario

Now that we know that most application framework will just accept request coming directly from a reverse proxy, let's see what we can actually achieve.

Proxy IP address spoofing

This is pretty obvious. As long as you can send requests on behalf of the Apache reverse proxy, you are automatically spoofing its IP address. This is particularly interesting when both the application and Apache reverse proxy are hosted on the same machine. In that case, the request to the application will appear to come from localhost.

Performing IP based access control, while whitelisting localhost as a trusted address, is not uncommon. It is even what allowed us to identify the mod_proxy issue in the first place during an intrusion test.

A quick dumb search through open source projects can already lead to some applications that would be affected when set behind an Apache reverse-proxy.

https://grep.app/search?q=%5C.remote_addr%20%3F%3D%3D%20%3F%5B%27%22%5D127&regexp=true&filter[lang][0]=Python

The typical code snippet that you would look for is, for instance, demonstrated in the Ziconius/FudgeC2 project returned by previous search.

def shutdown_listener():
    if request.remote_addr == "127.0.0.1":
        shutdown_hook = request.environ.get('werkzeug.server.shutdown')
        if shutdown_hook is not None:
            shutdown_hook()

In that case, having an Apache reverse proxy in front of the app would lead to an authentication bypass.

Arbitrary address spoofing

In some situations, it might be possible to extend the previous attack a bit. Particularly, if the application or framework accepts other sources for IP addresses as backup for X-Forwarded-For. In that case, forcing the X-Forwarded-For header to be removed while adding an alternative could allow spoofing an arbitrary IP address.

This is the case with a default configuration of django-ipware when the IPWARE_META_PRECEDENCE_ORDER setting is left untouched. Depending on the other settings, removing the X-Forwarded-For header might lead either to no IP address or the proxy one being returned by the component.

$ curl -i 192.168.122.225/test/ip/
HTTP/1.1 200 OK
[SKIPPED]

Hello 192.168.122.1
$ curl -H "Connection: close, X-Forwarded-For" -i 192.168.122.225/test/ip/
HTTP/1.1 200 OK
[SKIPPED]

Hello 127.0.0.1

But in that case, spoofing an arbitrary IP becomes possible by adding a Client-IP header to the request.

$ curl -H "Client-IP: 10.10.10.10" -H "Connection: close, X-Forwarded-For" -i 192.168.122.225/test/ip/
HTTP/1.1 200 OK
Date: Thu, 07 Jul 2022 14:34:50 GMT
Server: WSGIServer/0.2 CPython/3.9.2
Content-Type: text/html; charset=utf-8
X-Frame-Options: DENY
Content-Length: 17
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Cross-Origin-Opener-Policy: same-origin
Connection: close

Hello 10.10.10.10

 

Host injection / host filter bypass

So far we only worked with the X-Forwarded-For header to achieve some kind of IP address spoofing. X-Forwarded-Host can also be a good attack option. Particularly in complex architectures, it might be possible to access the default VirtualHosts of backend servers or applications while those are not intended to be accessed from public access.

In fact, what is pretty funny is that this exploitation scenario has already been used in the wild, as part of a bigger exploit chain. Related vulnerability is CVE-2022-1388, an authentication bypass to remote code execution in F5 BIG-IP, which you have probably heard about.

If you read the detailed explanation about this issue [F5VULN], you might notice the following statement:

  • Connection: X-F5-Auth-Token, X-Forwarded-Host

This prohibits Jetty from knowing that the request was provided by Apache and treats the request as if it were done locally.

And this is actually the issue we are discussing. By removing the X-Forwarded-Host header, the jetty server is tricked into considering the request as local, allowing access to the administration application.

Denial of service

Because we can send requests on behalf of the reverse proxy, we could try to abuse rate limiting or anti-bruteforce mechanisms. For example, imagine a fail2ban protection set on failed authentications from an IP address. Such a protection could be abused to force the application server to blacklist the reverse proxy's IP address.

Wrapping up

Apache HTTPD mod_proxy between versions 2.2.1 and 2.4.53, does not fill the X-Forwarded headers when those are listed as hop-by-hop. Applications hosted behind it can misunderstand the real client's IP address or requested hostname.

Depending on the applications and architectures, this can lead to an authentication or filtering bypass, an IP address spoofing or a denial of service. This is illustrated by vulnerability CVE-2022-1388, which exploit uses mod_proxy's issue to access a private application.

The issue is fixed in Apache HTTPD version 2.4.54.

Timeline

Time Event
05/10 Issue reported to HTTPD security team.
05/18 Issue is refused by HTTPD security team.
05/18 Further details added to the case.
05/30 Issue accepted.
06/08 Patch released.
06/09 CVE published.