Running Kubernetes on OCI Free Tier (Part 3): Networking Deep Dive

This is the most technical post in the series. In Part 1 we stood up the OCI infrastructure with Terraform, and Part 2 covered multi-cluster GitOps with Flux. Now we’re going deep on networking — the layer where the homelab and OCI clusters diverge the most.

Running two clusters with fundamentally different CNIs (Cilium eBPF vs Flannel iptables) has been a great exercise in understanding what the networking layer actually does for you, and what you lose when you can’t choose it.

Why Cilium Isn’t Possible on OCI Free Tier

This was the first thing I investigated when setting up OKE, and the answer was disappointing but clear.

OKE BASIC_CLUSTER (the free tier) only supports Flannel Overlay as the CNI. It’s hardcoded by Oracle. The cluster lifecycle manages the CNI installation, and you cannot replace it.

OKE ENHANCED_CLUSTER supports VCN-Native Pod Networking where pods get IPs directly from the OCI VCN subnet, but that’s Oracle’s own CNI — still not Cilium. And enhanced clusters cost money, which defeats the purpose.

To run Cilium on OCI you’d need self-managed VMs (Talos or kubeadm on Compute instances) where you control the CNI bootstrap. But that means managing the control plane yourself, losing OKE’s managed upgrades, and burning Always Free compute quota on control plane nodes. Not practical.

So: Flannel + kube-proxy (iptables) is the only option on OKE free tier. Not ideal, but it works, and the constraints make for a good comparison.

North-South Traffic: How External Traffic Enters Each Cluster

This is where the architectural differences get interesting. Both clusters use Cloudflare Tunnels for external access, but the routing patterns are completely different.

Homelab: Dual Ingress Paths

The homelab has two parallel L7 routing surfaces operating simultaneously.

Path A — External/Internet Services (Cloudflare Tunnel + MetalLB)

Internet user
  -> Cloudflare DNS (CNAME -> e56940bd.cfargotunnel.com)
  -> Cloudflare edge PoP (TLS terminated by Cloudflare)
  -> Encrypted QUIC/HTTP2 tunnel (outbound from cloudflared DaemonSet)
  -> Per-hostname routing in tunnel config:
      grafana.xjohnyx.me -> prometheus-stack-grafana.monitoring.svc:80 (cluster DNS)
      mattermost.xjohnyx.me -> 10.11.3.29:8065 (MetalLB LB IP)
  -> Cilium eBPF DNAT resolves ClusterIP/LB IP -> pod IP
  -> Pod

Each external service has an explicit entry in the cloudflared values.yaml with a specific backend — either an in-cluster DNS name or a MetalLB LoadBalancer IP. Adding a new external service requires updating the tunnel config.

MetalLB operates in L2 mode (ARP/NDP advertisement):

• IP pool: 10.11.3.1-10.11.3.254 (254 addresses)
• One leader node answers ARP for each virtual IP
• No ECMP — single-node ingress per IP, failover on leader re-election
• IP assignment is instantaneous (no cloud API calls)

Some services also define Gateway API HTTPRoute resources that attach to an external Gateway, giving them a second routing path via Cilium’s Gateway API controller.

Path B — Internal/Private Services (Cilium Ingress + cert-manager)

LAN browser
  -> DNS: service-internal.xjohnyx.me -> A record -> Cilium shared Ingress LB IP
  -> Cilium Ingress Controller (shared LB, eBPF XDP/TC hooks)
  -> TLS terminated with Let's Encrypt cert (DNS-01 via Cloudflare API)
  -> HTTP routed to backend Service -> Pod

Cilium’s built-in Ingress controller runs in shared mode — all Ingress objects share a single MetalLB LoadBalancer IP. TLS certificates come from cert-manager using DNS-01 challenges, which works behind firewalls since validation happens via Cloudflare DNS TXT records, not HTTP.

ExternalDNS on the homelab watches three source types: gateway-httproute, ingress, and service — simultaneously creating DNS records from all three Kubernetes API surfaces.

OCI: Single Wildcard Tunnel -> nginx

The OCI approach is simpler by necessity.

Internet user
  -> Cloudflare DNS (CNAME -> oci-lab-tunnel.cfargotunnel.com)
  -> Cloudflare edge PoP
  -> oci-lab-tunnel (cloudflared in cloudflare-system namespace)
  -> Wildcard rule: *.xjohnyx.me -> http://ingress-nginx-controller.ingress-nginx.svc:80
  -> kube-proxy iptables DNAT -> nginx pod
  -> nginx L7 routing by Host header -> backend Service
  -> kube-proxy iptables DNAT -> backend Pod

The key architectural difference: OCI uses a single wildcard *.xjohnyx.me tunnel rule pointing to nginx-ingress. ALL L7 routing decisions happen inside nginx based on Host headers. Adding a new service only requires creating a Kubernetes Ingress object — the tunnel config never changes.

The homelab uses per-hostname tunnel rules with specific backends. This gives finer-grained control (routing Grafana via cluster DNS vs Mattermost via MetalLB IP) but requires a tunnel config update for every new external service.

The OCI Load Balancer is a real cloud-managed resource:

• Provisioned by OCI Cloud Controller Manager in the public subnet
• Has its own public IP, health checks, and availability
• Flexible shape: 10 Mbps (free tier cap)
• Takes 1-3 minutes to provision (vs MetalLB’s instant ARP assignment)
• Annotations: oci-load-balancer-shape: flexible, flex-min/max: 10 Mbps

The wildcard tunnel pattern is honestly something I’d consider adopting on the homelab too. The per-hostname approach gives you control, but for most services, you just want “route to nginx and let Ingress objects handle it.”

East-West Traffic: Pod-to-Pod Communication

This is where the CNI choice has the biggest impact. The difference between Cilium eBPF and Flannel iptables isn’t just academic — you can feel it in how the cluster behaves.

Homelab: Cilium eBPF (Zero iptables)

With kubeProxyReplacement: true, Cilium eliminates iptables entirely from the data path:

Pod A connects to ClusterIP:port
  -> Cilium sockLB (BPF hook at connect() syscall)
  -> Destination rewritten to pod IP BEFORE packet is built
  -> If same node: direct veth delivery (no network stack traversal)
  -> If remote node: kernel routing table -> direct routing or tunnel
  -> Cilium BPF at destination veth enforces NetworkPolicy (identity-based)
  -> Pod B receives packet

The performance characteristics here are significant:

• Socket-level load balancing (sockLB): intercepts the connect() syscall and rewrites the destination before the TCP SYN is even constructed. No conntrack entry needed for the service VIP.
• No VXLAN overhead: Cilium in ipam.mode: kubernetes uses direct routing (kernel routes per pod CIDR) when the underlying network supports it. Zero per-packet encapsulation.
• O(1) NetworkPolicy: enforced via eBPF hash maps on each pod’s veth. Policy lookup is constant-time regardless of rule count.
• No iptables rules at all: zero NAT chains, zero conntrack overhead for east-west traffic.

OCI: Flannel + kube-proxy (iptables + VXLAN)

Pod A connects to ClusterIP:port
  -> Packet enters kernel network stack
  -> iptables PREROUTING chain (kube-proxy rules)
  -> DNAT: random endpoint selected from iptables probability rules
  -> conntrack entry created for NAT state
  -> If remote node: Flannel VXLAN encapsulation (50 bytes overhead)
  -> UDP packet traverses OCI VCN between nodes
  -> Remote node: VXLAN decap -> packet delivered to pod veth
  -> Pod B receives packet

The thing that surprised me most here wasn’t the performance — it’s a small cluster, and the overhead is tolerable. It’s the lack of visibility. With Flannel:

• O(n) iptables traversal: kube-proxy writes one iptables rule per Service endpoint. With 50+ Services, hundreds or thousands of rules are evaluated linearly on every new connection.
• VXLAN overhead: every cross-node packet gets 50 bytes of encapsulation (outer Ethernet + IP + UDP + VXLAN headers) plus CPU cost of encap/decap.
• Conntrack for every connection: stateful NAT tracking for every Service connection, consuming kernel memory and CPU.
• No NetworkPolicy enforcement: Flannel has no NetworkPolicy support. Every pod can talk to every pod. You’d need Calico or another policy controller layered on top.

For a two-node cluster running monitoring workloads, this is fine. But it reinforces why I run Cilium on the homelab — once you’ve had eBPF-based networking, going back to iptables feels like going back to awk after learning jq.

Gateway API vs Classic Ingress

Homelab: Cilium Gateway API

The homelab runs Cilium’s built-in Gateway API controller with two Gateways:

• External Gateway — ports 80/443, wildcard TLS, accepts routes from all namespaces
• Internal Gateway — port 80 only, LAN traffic

Services attach via HTTPRoute resources that reference a parent Gateway. Cilium compiles these routes into eBPF maps — no Envoy sidecar involved.

The homelab simultaneously supports three routing mechanisms:

1. Gateway API HTTPRoute (modern, cross-namespace routing)
2. Classic Ingress with ingressClassName: cilium (legacy compatibility)
3. Annotated Service (ExternalDNS creates records from service annotations)

This flexibility is one of the things I appreciate about Cilium’s approach. You can incrementally adopt Gateway API without ripping out existing Ingress objects.

OCI: Classic Ingress Only

OCI has no GatewayClass, no Gateway, no HTTPRoute. All L7 routing uses standard networking.k8s.io/v1 Ingress objects processed by nginx-ingress (set as default IngressClass).

ExternalDNS on OCI watches only ingress and service sources — no gateway-httproute because there are none.

It’s simpler, and honestly it works well for the OCI cluster’s workload. Gateway API shines when you have many teams sharing a cluster and need cross-namespace routing with RBAC boundaries. For a two-service monitoring cluster, classic Ingress is the right call.

Network Observability

This is the section that makes me most grateful for Cilium on the homelab.

Homelab: Hubble

Cilium includes Hubble — an eBPF-native observability layer:

• Taps into eBPF ring buffers on every node
• Per-flow visibility at L3/L4/L7 (HTTP method, URL, gRPC, DNS queries)
• Service dependency maps
• Network policy verdict logging (allow/deny per flow)
• No traffic mirroring, no sidecars, no performance overhead

I have it exposed at hubble-internal.xjohnyx.me behind the Cilium Ingress, and it’s one of the first things I check when debugging connectivity issues. Being able to see every DNS query, every HTTP request, and every policy verdict — all from eBPF ring buffers with zero overhead — is the kind of thing you don’t appreciate until you don’t have it.

OCI: No Network Observability

Flannel provides zero flow-level visibility. If a pod can’t reach a service on OCI, you’re back to kubectl exec and curl to figure out what’s happening. You could layer on OCI Flow Logs at the VCN level, but those show VCN-level traffic, not pod-level. A service mesh (Istio, Linkerd) would give you L7 visibility, but that’s a lot of overhead for a two-node free tier cluster.

Consolidated Comparison Table

Here’s everything side by side. This table covers networking, but also the broader infrastructure differences since they inform the networking choices:

Dimension	Homelab (Proxmox/Talos)	OCI (OKE Free Tier)
CNI	Cilium 1.18.6 (eBPF)	Flannel Overlay (VXLAN)
kube-proxy	Fully replaced by Cilium eBPF	Standard iptables mode
Node arch	x86-64 (Talos Linux on Proxmox VMs)	ARM64 (Oracle Linux 8, Ampere A1)
External LB	MetalLB L2 (ARP, 10.11.3.0/24 pool)	OCI Managed LB (flexible 10 Mbps)
Ingress	Cilium built-in (shared LB mode)	nginx-ingress (default IngressClass)
Gateway API	Cilium GatewayClass + 2 Gateways + HTTPRoute	Not available
Tunnel pattern	Per-hostname -> specific LB IPs or cluster DNS	Wildcard *.xjohnyx.me -> nginx ClusterIP
TLS termination	Cilium Ingress (eBPF)	nginx (userspace)
East-west routing	sockLB (BPF at connect()), direct pod routing	iptables DNAT + conntrack + VXLAN
NetworkPolicy	Cilium eBPF (identity-based, O(1))	None (Flannel has no support)
Network observability	Hubble (eBPF flow logs, L3-L7)	None
ExternalDNS sources	gateway-httproute, ingress, service	ingress, service
Storage	Synology NAS NFS	OCI Block Volumes
Monitoring	Prometheus + Grafana + Loki + Tempo	Not deployed yet

What I’d Do Differently

If I were starting over, a few things I’d change on the networking side:

Adopt the wildcard tunnel pattern on the homelab. The per-hostname approach gives fine-grained control, but for 90% of services, it’s unnecessary complexity. A wildcard rule pointing to the Cilium Ingress would simplify onboarding new services.

Layer Calico on OCI for NetworkPolicy. Even on a small cluster, having zero network segmentation isn’t great practice. Calico can run alongside Flannel purely for policy enforcement without replacing the CNI.

Use Gateway API on OCI when nginx supports it. The nginx Gateway API implementation is maturing. When it’s stable, migrating OCI from Ingress to HTTPRoute would bring the two clusters closer together in configuration shape.

Wrapping Up

Running two clusters with different networking stacks has been more educational than I expected. The homelab with Cilium eBPF is the premium experience — sockLB, direct routing, Hubble observability, Gateway API, identity-based NetworkPolicy. The OCI cluster with Flannel is the “it works” experience — iptables, VXLAN, no observability, no policy enforcement.

Both serve their purpose. The OCI cluster monitors my homelab from Bogota, and it doesn’t need eBPF to do that well. But the comparison makes it clear why the Kubernetes ecosystem is moving toward eBPF-based networking. The gap between what Cilium gives you and what Flannel gives you isn’t a nice-to-have — it’s a fundamentally different operational experience.

If you’re running OKE on the free tier and want Cilium, your only real path is self-managed nodes with Talos or kubeadm. Whether that tradeoff is worth it depends on your workload. For a monitoring satellite cluster, Flannel is fine. For anything with real traffic and security requirements, you’ll want control over your CNI.

---

This is Part 3 of 3. Part 1 covers infrastructure setup with Terraform. Part 2 covers multi-cluster GitOps with Flux CD.

Infrastructure: Terraform + OCI Provider | GitOps: Flux CD v2.7 | Kubernetes: OKE v1.32 on ARM64 | Secrets: SOPS + Age | CNI: Cilium (homelab) / Flannel (OCI)