Background
My infra is on AWS. My services need to access different third party APIs. These APIs have strict IP whitelist rules and rate limits. My services have to access these APIs with specific whitelisted IPs. Thus I need to either 1. run corresponding services on dedicated servers with reserved IPs, or 2. implement a proxy layer responsible for proxying my services’ requests.
Option 1 is obviously inflexible and not scalable. I could have many micro-services on one instance. Sometimes even for the same third party API, I may need to access it with different static IPs because I am using multiple API keys and each API key associated with a different whitelisted IP.
No the problem is, how to implement option 2 proxy?
Multi-Public IP Outbound EC2
In this example, I want to associate 3 elastic IPs on one EC2 instance so it can send outbound requests with these public IPs as source IP.
Normally this will require spawning 3 EC2 instances and assigning public IPs to them, or using AWS load balancers like NLB, ALB.
But since we don’t really need to care about inbound traffic, our stack could be simplified to just one EC2. This saves so much complexity and cost.
Here is my Pulumi config.
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
// Configure AWS region
const config = new pulumi.Config();
const awsConfig = new aws.Provider("aws", {
region: "ap-east-1",
});
// Create 3 Elastic IPs
const eip1 = new aws.ec2.Eip("eip1", {}, { provider: awsConfig });
const eip2 = new aws.ec2.Eip("eip2", {}, { provider: awsConfig });
const eip3 = new aws.ec2.Eip("eip3", {}, { provider: awsConfig });
// Create a VPC and subnet for the EC2 instance
const vpc = new aws.ec2.Vpc(
"vpc",
{
cidrBlock: "10.0.0.0/16",
enableDnsHostnames: true,
enableDnsSupport: true,
},
{ provider: awsConfig }
);
const subnet = new aws.ec2.Subnet(
"subnet",
{
vpcId: vpc.id,
cidrBlock: "10.0.1.0/24",
availabilityZone: "ap-east-1a",
},
{ provider: awsConfig }
);
const internetGateway = new aws.ec2.InternetGateway(
"internetGateway",
{
vpcId: vpc.id,
},
{ provider: awsConfig }
);
const routeTable = new aws.ec2.RouteTable(
"routeTable",
{
vpcId: vpc.id,
routes: [
{
cidrBlock: "0.0.0.0/0",
gatewayId: internetGateway.id,
},
],
},
{ provider: awsConfig }
);
const routeTableAssociation = new aws.ec2.RouteTableAssociation(
"routeTableAssociation",
{
subnetId: subnet.id,
routeTableId: routeTable.id,
},
{ provider: awsConfig }
);
// Create a security group
const securityGroup = new aws.ec2.SecurityGroup(
"securityGroup",
{
vpcId: vpc.id,
ingress: [
{
protocol: "tcp",
fromPort: 22,
toPort: 22,
cidrBlocks: ["0.0.0.0/0"],
},
],
egress: [
{
protocol: "-1",
fromPort: 0,
toPort: 0,
cidrBlocks: ["0.0.0.0/0"],
},
],
},
{ provider: awsConfig }
);
// Create a network interface with multiple private IPs
const networkInterface = new aws.ec2.NetworkInterface(
"networkInterface",
{
subnetId: subnet.id,
privateIps: ["10.0.1.10", "10.0.1.11", "10.0.1.12"],
securityGroups: [securityGroup.id],
},
{ provider: awsConfig }
);
// Create an EC2 instance with the network interface
const instance = new aws.ec2.Instance(
"instance",
{
ami: "ami-050f19c6ee04f419b", // Amazon Linux 2 AMI
instanceType: "t3.medium",
networkInterfaces: [
{
networkInterfaceId: networkInterface.id,
deviceIndex: 0,
},
],
keyName: "bitlake-hk",
tags: {
Name: "pulumi-instance",
},
},
{ provider: awsConfig }
);
// Associate Elastic IPs with the private IPs
const eipAssociation1 = new aws.ec2.EipAssociation(
"eipAssociation1",
{
networkInterfaceId: networkInterface.id,
privateIpAddress: "10.0.1.10",
allocationId: eip1.id,
},
{ provider: awsConfig }
);
const eipAssociation2 = new aws.ec2.EipAssociation(
"eipAssociation2",
{
networkInterfaceId: networkInterface.id,
privateIpAddress: "10.0.1.11",
allocationId: eip2.id,
},
{ provider: awsConfig }
);
const eipAssociation3 = new aws.ec2.EipAssociation(
"eipAssociation3",
{
networkInterfaceId: networkInterface.id,
privateIpAddress: "10.0.1.12",
allocationId: eip3.id,
},
{ provider: awsConfig }
);
// Export the instance's public IP and the Elastic IPs
export const instancePublicIp = instance.publicIp;
export const elasticIp1 = eip1.publicIp;
export const elasticIp2 = eip2.publicIp;
export const elasticIp3 = eip3.publicIp;2 secondary private IPs are created, and associated with Elastic IPs. Then when I send request out, specify an interface (private IP) to use and it will use the corresponding public IP to send out requests.
// # Test first IP
// curl --interface 10.0.1.10 http://example.com
// # Test second IP
// curl --interface 10.0.1.11 http://example.com
// # Test third IP
// curl --interface 10.0.1.12 http://example.comThen I can write a custom proxy server with nginx or pingora and use the corresponding interface.
Nginx Deployment with ECS on EC2 with 3 Elastic IPs
In the following example, I demonstrate how to implement nginx proxy with 3 Elastic IPs.
In my own use case, the reverse proxy is used by internal services to proxy to other APIs with different source IPs. So SSL cert isn’t needed.
In the example, I will use let nginx proxy_pass to webhook.site so we can see what the source IP is.
The code explains itself, detailed comments are added.
Basically when nginx proxy_pass, it picks a specific network interface, like curl --interface 10.0.1.10.
Pulumi Definition
In this example, we are using the official nginx:latest image, and using userData to pass nginx.conf. However, in production deployment, if you modify nginx.conf and deploy again with pulumi up, userData will not run again, thus nginx.conf updates won’t be applied.
Consider build a custom docker image containing the conf and deploy it.
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";
import * as fs from "fs";
// Read the nginx configuration file
const nginxConfig = fs.readFileSync("nginx.conf", "utf8");
/**
* REVERSE PROXY WITH MULTIPLE SOURCE IPs
*
* This infrastructure creates a reverse proxy running on ECS that can forward
* requests to external services using different source IP addresses.
*
* Architecture:
* - Single EC2 instance with multiple private IPs (10.0.1.10, 10.0.1.11, 10.0.1.12)
* - Multiple Elastic IPs associated with each private IP
* - HAProxy container routing requests by path and using different source IPs
* - ECS managing the HAProxy container lifecycle
*
* Usage:
* - GET /ip1/* → Forward using first public IP
* - GET /ip2/* → Forward using second public IP
* - GET /ip3/* → Forward using third public IP
*/
// ============================================================================
// ELASTIC IPs (Multiple Source IPs for Proxy)
// ============================================================================
/**
* ELASTIC IPs: Three static public IP addresses for our proxy
*
* Why multiple IPs: Allows the proxy to appear as different sources to destination servers
* Use cases: Rate limiting bypass, geo-distribution simulation, load spreading
*/
const elasticIp1 = new aws.ec2.Eip("proxy-eip-1", {
domain: "vpc",
tags: {
Name: "proxy-ip-1",
Purpose: "reverse-proxy-source",
},
});
const elasticIp2 = new aws.ec2.Eip("proxy-eip-2", {
domain: "vpc",
tags: {
Name: "proxy-ip-2",
Purpose: "reverse-proxy-source",
},
});
const elasticIp3 = new aws.ec2.Eip("proxy-eip-3", {
domain: "vpc",
tags: {
Name: "proxy-ip-3",
Purpose: "reverse-proxy-source",
},
});
// ============================================================================
// NETWORKING (Custom VPC for Multiple IP Support)
// ============================================================================
/**
* CUSTOM VPC: Required for multiple IP configuration
*
* Why custom VPC: Default VPC doesn't support advanced networking features we need
* Features needed: Multiple private IPs per instance, Elastic IP associations
*/
const vpc = new aws.ec2.Vpc("proxy-vpc", {
cidrBlock: "10.0.0.0/16",
enableDnsHostnames: true,
enableDnsSupport: true,
tags: {
Name: "proxy-vpc",
},
});
/**
* PUBLIC SUBNET: Where our proxy instance will live
* Must be public so Elastic IPs can be associated
*/
const publicSubnet = new aws.ec2.Subnet("proxy-subnet", {
vpcId: vpc.id,
cidrBlock: "10.0.1.0/24",
availabilityZone: "ap-east-1a", // TODO: Make this dynamic like minimal-ecs.ts
mapPublicIpOnLaunch: true,
tags: {
Name: "proxy-public-subnet",
},
});
/**
* INTERNET GATEWAY: Required for public internet access
*/
const internetGateway = new aws.ec2.InternetGateway("proxy-igw", {
vpcId: vpc.id,
tags: {
Name: "proxy-igw",
},
});
/**
* ROUTE TABLE: Routes traffic to internet gateway
*/
const routeTable = new aws.ec2.RouteTable("proxy-routes", {
vpcId: vpc.id,
routes: [
{
cidrBlock: "0.0.0.0/0",
gatewayId: internetGateway.id,
},
],
tags: {
Name: "proxy-route-table",
},
});
/**
* ROUTE TABLE ASSOCIATION: Links subnet to route table
*/
const routeTableAssociation = new aws.ec2.RouteTableAssociation(
"proxy-rt-assoc",
{
subnetId: publicSubnet.id,
routeTableId: routeTable.id,
}
);
// ============================================================================
// SECURITY GROUP (Network Access Rules)
// ============================================================================
/**
* SECURITY GROUP: Firewall rules for the proxy instance
*
* Ports opened:
* - 80: HTTP access to proxy
* - 8080: HAProxy stats interface (optional)
* - 22: SSH access for debugging
*/
const securityGroup = new aws.ec2.SecurityGroup("proxy-sg", {
vpcId: vpc.id,
description: "Security group for reverse proxy instance",
ingress: [
{
protocol: "tcp",
fromPort: 80,
toPort: 80,
cidrBlocks: ["0.0.0.0/0"],
description: "HTTP access to proxy",
},
{
protocol: "tcp",
fromPort: 8080,
toPort: 8080,
cidrBlocks: ["0.0.0.0/0"],
description: "HAProxy stats interface",
},
{
protocol: "tcp",
fromPort: 22,
toPort: 22,
cidrBlocks: ["0.0.0.0/0"],
description: "SSH access for debugging",
},
],
egress: [
{
protocol: "-1",
fromPort: 0,
toPort: 0,
cidrBlocks: ["0.0.0.0/0"],
description: "All outbound traffic (for proxy forwarding)",
},
],
tags: {
Name: "proxy-security-group",
},
});
// ============================================================================
// NETWORK INTERFACE (Multiple Private IPs)
// ============================================================================
/**
* NETWORK INTERFACE: Custom network interface with multiple private IPs
*
* Why custom interface: Standard EC2 instances get one IP, we need three
* Private IPs: 10.0.1.10, 10.0.1.11, 10.0.1.12 (will map to public Elastic IPs)
* HAProxy will bind to these IPs and use them as source IPs for outbound requests
*/
const networkInterface = new aws.ec2.NetworkInterface("proxy-eni", {
subnetId: publicSubnet.id,
privateIps: ["10.0.1.10", "10.0.1.11", "10.0.1.12"],
securityGroups: [securityGroup.id],
tags: {
Name: "proxy-network-interface",
},
});
// ============================================================================
// ECS CLUSTER (Container Orchestration)
// ============================================================================
/**
* ECS CLUSTER: Manages our HAProxy container
* Same as minimal setup but for proxy instead of nginx
*/
const ecsCluster = new aws.ecs.Cluster("proxy-cluster", {
name: "reverse-proxy-cluster",
settings: [
{
name: "containerInsights",
value: "enabled",
},
],
tags: {
Name: "reverse-proxy-cluster",
},
});
// ============================================================================
// IAM ROLES (Permissions for ECS)
// ============================================================================
/**
* EC2 INSTANCE IAM ROLE: Allows EC2 to join ECS cluster
* Same role structure as minimal-ecs.ts (proven to work)
*/
const ec2Role = new aws.iam.Role("proxy-ec2-role", {
assumeRolePolicy: JSON.stringify({
Version: "2012-10-17",
Statement: [
{
Action: "sts:AssumeRole",
Effect: "Allow",
Principal: {
Service: "ec2.amazonaws.com",
},
},
],
}),
tags: {
Name: "proxy-ec2-role",
},
});
/**
* ECS POLICY ATTACHMENT: Required permissions for ECS agent
*/
const ec2RolePolicyAttachment = new aws.iam.RolePolicyAttachment(
"proxy-ec2-policy",
{
role: ec2Role.name,
policyArn:
"arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role",
}
);
/**
* CLOUDWATCH LOGS POLICY: For container logging
*/
const cloudWatchPolicyAttachment = new aws.iam.RolePolicyAttachment(
"proxy-cloudwatch-policy",
{
role: ec2Role.name,
policyArn: "arn:aws:iam::aws:policy/CloudWatchLogsFullAccess",
}
);
/**
* INSTANCE PROFILE: Bridge between IAM role and EC2 instance
*/
const instanceProfile = new aws.iam.InstanceProfile("proxy-instance-profile", {
role: ec2Role.name,
});
// ============================================================================
// EC2 INSTANCE (Compute for HAProxy Container)
// ============================================================================
/**
* ECS-OPTIMIZED AMI: Pre-configured for running containers
*/
const ecsAmi = aws.ec2.getAmi({
mostRecent: true,
owners: ["amazon"],
filters: [
{
name: "name",
values: ["amzn2-ami-ecs-hvm-*-x86_64-ebs"],
},
],
});
/**
* EC2 INSTANCE: The compute resource running our proxy
*
* Key differences from minimal-ecs.ts:
* - Uses custom network interface (for multiple IPs)
* - Larger instance type (t3.medium for better networking performance)
* - Custom user data that creates HAProxy configuration files
*/
const proxyInstance = new aws.ec2.Instance(
"proxy-instance",
{
ami: ecsAmi.then((ami) => ami.id),
instanceType: "t3.medium", // Larger for better network performance
keyName: "bitlake-hk",
networkInterfaces: [
{
networkInterfaceId: networkInterface.id,
deviceIndex: 0,
},
],
iamInstanceProfile: instanceProfile.name,
userData: pulumi.interpolate`#!/bin/bash
echo ECS_CLUSTER=${ecsCluster.name} >> /etc/ecs/ecs.config
# Wait for network interfaces to be fully configured
sleep 30
# Create directory for nginx config (ensure it exists)
mkdir -p /etc/nginx
# Write the nginx configuration file
cat > /etc/nginx/nginx.conf << 'EOF'
${nginxConfig}
EOF
# Log completion
echo "Nginx proxy setup completed at $(date)" >> /var/log/proxy-setup.log
echo "ECS_CLUSTER=${ecsCluster.name}" >> /var/log/proxy-setup.log
echo "Available IPs:" >> /var/log/proxy-setup.log
ip addr show >> /var/log/proxy-setup.log
`,
tags: {
Name: "reverse-proxy-instance",
Purpose: "multi-ip-proxy",
},
},
{
dependsOn: [instanceProfile, ec2RolePolicyAttachment, networkInterface],
}
);
// ============================================================================
// ELASTIC IP ASSOCIATIONS (Map Public IPs to Private IPs)
// ============================================================================
/**
* EIP ASSOCIATIONS: Link each Elastic IP to a private IP on the instance
*
* This creates the mapping:
* - Elastic IP 1 → 10.0.1.10 (accessible via /ip1/*)
* - Elastic IP 2 → 10.0.1.11 (accessible via /ip2/*)
* - Elastic IP 3 → 10.0.1.12 (accessible via /ip3/*)
*/
const eipAssociation1 = new aws.ec2.EipAssociation(
"proxy-eip-assoc-1",
{
networkInterfaceId: networkInterface.id,
privateIpAddress: "10.0.1.10",
allocationId: elasticIp1.id,
},
{ dependsOn: [proxyInstance] }
);
const eipAssociation2 = new aws.ec2.EipAssociation(
"proxy-eip-assoc-2",
{
networkInterfaceId: networkInterface.id,
privateIpAddress: "10.0.1.11",
allocationId: elasticIp2.id,
},
{ dependsOn: [proxyInstance] }
);
const eipAssociation3 = new aws.ec2.EipAssociation(
"proxy-eip-assoc-3",
{
networkInterfaceId: networkInterface.id,
privateIpAddress: "10.0.1.12",
allocationId: elasticIp3.id,
},
{ dependsOn: [proxyInstance] }
);
// ============================================================================
// ECS TASK EXECUTION ROLE (For Container Management)
// ============================================================================
/**
* TASK EXECUTION ROLE: Permissions for ECS to manage HAProxy container
*/
const taskExecutionRole = new aws.iam.Role("proxy-task-execution-role", {
assumeRolePolicy: JSON.stringify({
Version: "2012-10-17",
Statement: [
{
Action: "sts:AssumeRole",
Effect: "Allow",
Principal: {
Service: "ecs-tasks.amazonaws.com",
},
},
],
}),
});
const taskExecutionRolePolicyAttachment = new aws.iam.RolePolicyAttachment(
"proxy-task-execution-policy",
{
role: taskExecutionRole.name,
policyArn:
"arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy",
}
);
// ============================================================================
// CLOUDWATCH LOGS (Container Logging)
// ============================================================================
/**
* CLOUDWATCH LOG GROUP: Centralized logging for HAProxy container
*/
const logGroup = new aws.cloudwatch.LogGroup("proxy-logs", {
name: "/ecs/reverse-proxy",
retentionInDays: 7,
tags: {
Name: "reverse-proxy-logs",
},
});
// ============================================================================
// ECS TASK DEFINITION (HAProxy Container Blueprint)
// ============================================================================
/**
* TASK DEFINITION: Defines how to run the HAProxy container
*
* Key configuration:
* - networkMode: "host" (critical for accessing multiple IPs)
* - HAProxy 2.8 image (latest stable)
* - Host network allows container to bind to specific IPs
* - Volume mount for config file from instance
*/
const taskDefinition = new aws.ecs.TaskDefinition("proxy-task", {
family: "reverse-proxy",
requiresCompatibilities: ["EC2"],
networkMode: "host", // Required for proxy_bind to work with multiple IPs
executionRoleArn: taskExecutionRole.arn,
cpu: "512",
memory: "1024",
containerDefinitions: JSON.stringify([
{
name: "nginx",
image: "nginx:latest",
essential: true,
portMappings: [
{
containerPort: 80,
hostPort: 80,
protocol: "tcp",
},
],
mountPoints: [
{
sourceVolume: "nginx-config",
containerPath: "/etc/nginx/nginx.conf",
readOnly: true,
},
],
logConfiguration: {
logDriver: "awslogs",
options: {
"awslogs-group": "/ecs/reverse-proxy",
"awslogs-region": "ap-east-1",
"awslogs-stream-prefix": "nginx",
},
},
},
]),
volumes: [
{
name: "nginx-config",
hostPath: "/etc/nginx/nginx.conf",
},
],
tags: {
Name: "reverse-proxy-task",
},
});
// ============================================================================
// ECS SERVICE (Container Lifecycle Management)
// ============================================================================
/**
* ECS SERVICE: Ensures HAProxy container stays running
*
* Manages:
* - Keeps exactly 1 HAProxy container running
* - Restarts failed containers
* - Handles deployments and updates
*/
const proxyService = new aws.ecs.Service(
"proxy-service",
{
cluster: ecsCluster.id,
taskDefinition: taskDefinition.arn,
desiredCount: 1,
launchType: "EC2",
},
{
dependsOn: [proxyInstance, logGroup],
}
);
// ============================================================================
// OUTPUTS (Access Information)
// ============================================================================
/**
* OUTPUTS: Important information for using the proxy
*/
// The three public IPs that can be used for proxy access
export const proxyIp1 = elasticIp1.publicIp;
export const proxyIp2 = elasticIp2.publicIp;
export const proxyIp3 = elasticIp3.publicIp;
// URLs for testing the proxy with different source IPs
export const testUrl1 = elasticIp1.publicIp.apply(
(ip) => `http://${ip}/ip1/unique-test-id-123`
);
export const testUrl2 = elasticIp2.publicIp.apply(
(ip) => `http://${ip}/ip2/unique-test-id-456`
);
export const testUrl3 = elasticIp3.publicIp.apply(
(ip) => `http://${ip}/ip3/unique-test-id-789`
);
// ECS cluster information
export const clusterName = ecsCluster.name;
export const serviceName = proxyService.name;
/**
* USAGE INSTRUCTIONS:
*
* 1. Test different source IPs:
* curl http://<proxyIp1>/ip1/your-webhook-id # Uses source IP 1
* curl http://<proxyIp2>/ip2/your-webhook-id # Uses source IP 2
* curl http://<proxyIp3>/ip3/your-webhook-id # Uses source IP 3
*
* 2. Check logs:
* aws logs tail /ecs/reverse-proxy --follow
*
* 3. Update webhook URL:
* Edit the HAProxy config in the user data and redeploy
*
* ARCHITECTURE SUMMARY:
* - 3 Elastic IPs for different source identities
* - 1 EC2 instance with 3 private IPs
* - HAProxy container with host networking
* - Path-based routing (/ip1, /ip2, /ip3)
* - Each path uses different source IP for outbound requests
* - ECS manages container lifecycle and health
*/nginx.conf
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
# Server listening on all interfaces
server {
listen 80;
# Health check endpoint
location /health {
return 200 'OK\n';
add_header Content-Type text/plain;
}
# Proxy requests through IP1 (10.0.1.10 -> 43.199.238.137)
location /ip1/ {
# Use first IP as source for outbound requests
proxy_bind 10.0.1.10;
proxy_pass https://webhook.site/76a7ab15-af8a-40e2-ab91-962d06675a31/;
}
# Proxy requests through IP2 (10.0.1.11 -> 18.162.247.245)
location /ip2/ {
# Use second IP as source for outbound requests
proxy_bind 10.0.1.11;
proxy_pass https://webhook.site/76a7ab15-af8a-40e2-ab91-962d06675a31/;
}
# Proxy requests through IP3 (10.0.1.12 -> 43.198.116.164)
location /ip3/ {
# Use third IP as source for outbound requests
proxy_bind 10.0.1.12;
proxy_pass https://webhook.site/76a7ab15-af8a-40e2-ab91-962d06675a31/;
}
# Default location - show usage info
location / {
return 200 'Proxy is working';
add_header Content-Type text/plain;
}
}
}