I like self hosting, I enjoy managing my own infrastructure, I do not like posting my IP on the internet. Obviously, that creates some issues. Obfuscating your address is normally not too much of an issue, there are plenty of ways to do it, I utilize cloudflare zero-trust tunnels. I get the benefits from Cloduflare DNS, DDoS Protection, etc, and I get to utilize my stack from anywhere via the Cloudflared.

Initially I set everything up as E2Edge, I would terminate my TLS at the edge then just navigate internally with plain HTTP; not the most secure, but also not a big deal with the way my architecture. Here's to you, Authentik! Genuinely one of the better services I've deployed.

This setup worked fine, but as I grew I realized there are genuine benefits to hosting your own container registry - you can utilize intra-cluster speeds instead of ISP bandwidth which can often be painfully slow when uploading containers. By utilizing intracluster communication I was able to increase the speed of the uploads and downloads of my containers with a hairpin DNS setup via coredns, but I still wanted to be able to externally access these containers for remote services or allowing friends of mine to utilize things I've built.

Sounds reasonable enough?

The problem: Gitlab container registry pushing expects HTTPS

Registry is set up, I'm authenticated, image pulls work, image pushes... fail?

$ podman push gitlab-registry.example.com/project/image:latest
Error: authenticating creds for "gitlab-registry.example.com":
  authentication required

I know it's authenticated. I can pull containers that require authentication? What's going on here.

The Debugging Spiral

I chased a few wrong things, validating the registry configuration, testing ingress timeouts, confirming the pumas are working. Who knew there were so many large cats in a gitlab stack.

Now, Docker and Podman authenticate to registries using a challenge-response flow:

Client requests a push
Registry responds with 401 Unauthorized and a Www-Authenticate header pointing to a token endpoint
Client hits the token endpoint to get a bearer token
Client retries the push with the token

Taking a step back I captured the HTTPS exchange directly to diagnose from there and found that the registry was returning the token endpoint URL as http:// instead of https://. My client dutifully followed that HTTP URL, but Cloudflare (correctly!) redirected HTTP to HTTPS. The redirect broke the auth flow.

What's happening here?

GitLab's container registry generates URLs based on what it sees in the incoming request. When ingress-nginx forwarded the request over plain HTTP, the registry saw:

Protocol: http
Host: gitlab-registry.example.com

So it generated http://gitlab-registry.example.com/... in its responses.

The registry had no way to know the original request came in over HTTPS through Cloudflare. From its perspective, HTTP was correct.

The Workaround: X-Forwarded-Proto

The quick fix was telling the registry what protocol the client actually used. You can configure ingress-nginx to inject headers via annotations on the Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gitlab-registry
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      proxy_set_header X-Forwarded-Proto https;
spec:
  # ... rest of ingress config

This tells the registry: "Even though you received this over HTTP, the original client used HTTPS." The registry then generates HTTPS URLs in its responses.

This works, but if I stopped at "Working" I'd still be hosting individually managing a bunch of boxes directly instead of accidentally becoming qualified enough for a CKA certification.

The X-Forwarded-Proto approach has problems:

Every application needs to respect it. Some do, some don't, some need explicit configuration. I'd have to remember this for every new service.
It's lying to the application. The registry thinks it's serving HTTPS, but it's actually serving HTTP. If any component along the path doesn't properly forward the header, things break silently.
The traffic is actually plaintext. Between Cloudflare and my cluster, traffic was unencrypted. My ISP, anyone on my local network, or any compromised tunnel component could see it.
GitLab wasn't the only problem. I hit similar issues with other applications that generate URLs based on the incoming protocol. OAuth callbacks, webhook URLs, asset links - lots of things break when the protocol doesn't match expectations.

I wanted actual HTTPS all the way through. Not "pretend HTTPS via headers."

The Real Fix: TLS to the Origin

The solution was straightforward in concept: just go fully E2E

This meant I needed certificates for my internal services. Real ones, not self-signed - Cloudflare validates origin certificates, and I didn't want to deal with certificate trust issues.

What I Needed

Automated certificate issuance - I wasn't going to manually request certs for every service
Automated renewal - Let's Encrypt certs expire every 90 days
Works behind the tunnel - No public port 80 for HTTP-01 challenges (or so I thought)
Works for internal services - Some services aren't exposed through Cloudflare at all

The Solution: cert-manager with Multiple Issuers

cert-manager is a Kubernetes add-on that handles certificate lifecycle. You define "Issuers" that know how to get certificates, and cert-manager does the rest.

I set up two types of issuers for different use cases.

HTTP-01 for Public Services

HTTP-01 challenges work by Let's Encrypt requesting a specific URL on your domain. cert-manager creates a temporary pod to respond.

Here's the surprising thing: this works through Cloudflare Tunnel. When Let's Encrypt hits http://gitlab.example.com/.well-known/acme-challenge/<token>, Cloudflare proxies the request through my tunnel to ingress-nginx, which routes to cert-manager's solver pod.

I didn't need port 80 exposed on my router. The tunnel handles it.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-key
    solvers:
      - http01:
          ingress:
            class: nginx

Gotcha: This only works if Cloudflare is set to "Proxied" (orange cloud) mode. In DNS-only mode, traffic goes directly to whatever IP is in the record - which isn't your cluster.

DNS-01 for Internal Services and Wildcards

Some services aren't exposed through Cloudflare at all - internal admin UIs, monitoring dashboards, etc. HTTP-01 can't reach them.

DNS-01 solves this by proving domain ownership through DNS TXT records. cert-manager uses the Cloudflare API to create the records, Let's Encrypt verifies them, done. No HTTP traffic required.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cloudflare-prod
spec:
  acme:
    email: admin@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-cloudflare-key
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token
      selector:
        dnsZones:
          - "example.com"

DNS-01 also unlocks wildcard certificates (*.example.com), which HTTP-01 can't do.

For an internal service not exposed through Cloudflare, you'd use this issuer instead:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: internal-dashboard
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-cloudflare-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - dashboard.internal.example.com
      secretName: dashboard-tls
  # ...

Security note: The Cloudflare API token needs Zone:DNS:Edit permission. Scope it to only the zones you need.

Using Certificates in Ingresses

Once the issuers are configured, getting a certificate is one annotation:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gitlab
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - gitlab.example.com
      secretName: gitlab-tls
  rules:
    - host: gitlab.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gitlab
                port:
                  number: 8080

cert-manager sees the annotation, creates a Certificate resource, handles the ACME challenge, and stores the cert in the gitlab-tls secret. Renewals happen automatically!

Fixing GitLab

With TLS actually terminating at ingress-nginx, I could configure GitLab properly.

The gitlab.rb Configuration

# External URL is HTTPS (what users see)
external_url 'https://gitlab.example.com'

# But GitLab's internal nginx listens on HTTP
# because ingress-nginx handles TLS termination
nginx['listen_address'] = '0.0.0.0'
nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['redirect_http_to_https'] = false

# Registry also listens on HTTP internally
registry['enable'] = true
registry_external_url 'https://registry.gitlab.example.com'
registry['registry_http_addr'] = "0.0.0.0:5000"
registry_nginx['enable'] = false

This might look contradictory: external URLs are HTTPS, but nginx listens on HTTP. The key is that ingress-nginx terminates TLS before traffic reaches GitLab. GitLab sees HTTP requests from ingress-nginx, but the client's connection to the Ingress was actually HTTPS.

The difference from before: the hop between Cloudflare and ingress-nginx is now TLS instead of HTTP. When ingress-nginx sets X-Forwarded-Proto: https, it's telling the truth - the client really did connect over HTTPS. GitLab generates correct HTTPS URLs because the header accurately reflects the client's connection.

Registry Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: gitlab-registry
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-body-size: "0"  # unlimited for large images
    nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - gitlab-registry.example.com
      secretName: gitlab-registry-tls
  rules:
    - host: gitlab-registry.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gitlab-registry
                port:
                  number: 5000

After applying this, the auth failures were gone the registry now generates correct HTTPS URLs because the X-Forwarded-Proto header accurately reflects the client's connection.

This fixed my HTTPS issues, but I immediately stumbled into another issue with the way that Cloudflare truncates chunked transfers. Docker and Podman stream layer blobs using Transfer-Encoding: chunked and Cloudflare silently cuts them off mid-upload. I wrote about that separately in Debugging CloudFlare Chunked Upload Limits.

This isn't perfect end-to-end encryption - Cloudflare still terminates TLS at the edge and can inspect traffic. But it removes the class of bugs where applications generate wrong URLs because they think they're serving HTTP.

Troubleshooting: When Certificates Don't Issue

cert-manager is usually reliable, but when things go wrong, here's how to debug:

Check the Certificate Resource

kubectl describe certificate gitlab-tls -n gitlab

Look for status conditions. "Issuing certificate as Secret does not exist" is normal - it means cert-manager is working on it.

Check for Challenges

kubectl get challenges -A

If there's no Challenge resource, cert-manager couldn't match a solver. Check your ClusterIssuer's ingress class or DNS zone selector.

For HTTP-01: Verify the Challenge URL Works

curl -v http://gitlab.example.com/.well-known/acme-challenge/test

If this doesn't reach your cluster:

Is Cloudflare set to Proxied (orange cloud)?
Is the tunnel running?
Is the ingress class correct?

For DNS-01: Check the TXT Record

dig TXT _acme-challenge.gitlab.example.com

If the record doesn't exist, check cert-manager logs for Cloudflare API errors:

kubectl logs -n cert-manager deploy/cert-manager | grep -i cloudflare

tl;dr

1. "Trusted Internal HTTP" Creates Hidden Dependencies

The header-based workaround (X-Forwarded-Proto) worked, but it required every application to correctly handle the header. Some applications need explicit configuration. Some ignore it entirely. Each new service was a potential landmine.

Actual TLS to the origin eliminated this class of problems. Applications don't need special configuration to generate correct URLs - they just see HTTPS because that's what the connection actually is.

2. The Tunnel Doesn't Change HTTP-01

I assumed I'd need DNS-01 for everything because "port 80 isn't exposed." But Cloudflare Tunnel proxies HTTP just fine. The Let's Encrypt servers talk to Cloudflare's edge, not directly to my IP. HTTP-01 works for any service that's publicly accessible through the tunnel.

I still use DNS-01 for internal services and as a backup, but HTTP-01 handles most cases.

3. cert-manager's IngressShim is Worth Using

I started by creating Certificate resources manually - separate YAML files for each service. This was tedious and error-prone.

IngressShim lets you just annotate the Ingress with cert-manager.io/cluster-issuer: letsencrypt-prod. cert-manager automatically creates the Certificate resource. Adding TLS to a new service went from "copy Certificate YAML, edit names, apply" to "add one annotation."

4. The Registry Problem Would Have Happened Elsewhere

GitLab's registry was the first thing to break obviously, but the same issue - HTTP internally, HTTPS externally - affects anything that generates URLs:

OAuth callback URLs
Webhook endpoints
Asset URLs in HTML responses
API responses with Location headers

Fixing it at the transport layer (actual TLS) was more robust than fixing it application by application.

My Stack

For reference, here's what I'm running

Component	What I'm Using
Cluster	K3s
Certificate Management	cert-manager 1.x
Ingress Controller	ingress-nginx
Edge/Tunnel	Cloudflare Tunnel
DNS	Cloudflare
CA	Let's Encrypt
GitLab CE	Omnibus

Alison Alva