CloudConnector -- User Guide v1.0.0

This guide covers running CloudConnector: how a deployment is shaped, every input and output of the Terraform roots, how traffic is routed into the pipe, how it is sized and scaled, how it fails and recovers, how to watch it, and how to troubleshoot it. For a first deployment, start with the Quick Start; for what CloudConnector is, see the Overview.

Topology and deployment shapes

A pipe always has the same parts: an AWS relay fleet, a Google Cloud relay fleet, and one controller that both fleets register with under a single fleet id. Two Terraform shapes build them:

In the production shape the two roots are joined by one value: the pipe root's sensitive controller_side_tokens output is the controller root's pipe_side_tokens input. It carries each side's announce credential, so the side a relay is placed on and the side the controller recognises come from one definition.

Zones

Each end starts in one zone: <region>a on AWS and <region>-a on Google Cloud. A one-zone end has nowhere to place a replacement during a zone outage, so production deployments should use two or more:

• AWS. Every zone gets its own Auto Scaling group, its own GWLB endpoint and its own routes, and each zone runs at least one relay -- a zone without one would drop the traffic entering through its endpoint. Cross-zone load balancing is off, so the load balancer adds no inter-zone hop. The AWS end therefore rounds its relay count up across zones: a target of 5 relays over 3 zones runs 6. Set aws_network_azs (created networks) or aws_zone_subnets (your networks).
• Google Cloud. The managed instance group is regional and the internal load balancer is regional, so the group spreads exactly fleet_size relays over the zones you list in gcp_zones.

Relay addressing

Relays dial each other directly over the internet; the load balancers only steer traffic into the local relays. When it registers, a relay announces the address the far side should dial: its IPv6 global address if it has one, otherwise its public IPv4 address, otherwise its private address. A private address is not reachable from the other cloud, so every relay needs one of the first two:

• Google Cloud relays get an external address of the family set by pipe_external_addressing: ipv6 (the default, because both clouds bill public IPv4 per address per hour), ipv4 or dual. Created networks are dual-stack and have Cloud NAT, so relays without an external IPv4 can still reach IPv4 destinations.
• AWS relays sit in the zone subnets you give them and must be reachable by the Google Cloud relays and able to reach the controller, the Cloud Spectra licence service and the far side.

The two ends must share an address family the far side can reach.

Sides and the fleet id

Both ends register under one fleet id (fleet_id, which defaults to name). The ends are named aws and gcp (aws_side_id, gcp_side_id); these names must match the side ids in the pipe configuration, which the pipe root writes from the same variables.

Configuration reference

Only the inputs and outputs listed here are part of the interface. Every default shown is the code's.

The cloudconnector example

Its outputs are controller_url, aws_endpoint_service, gcp_ilb_ip, wg_port_range, relay_target_per_end, relay_count_aws, relay_count_gcp and relay_count, each described under the deployment root below. Any other input of the deployment root can be set by adding it to the example's module "cloudconnector" block.

Deployment root: cloudspectra-relay-deploy

Every input is optional.

Production pipe root: cloudspectra-relay-pipe

Builds both fleets, sizes them with cloudspectra_ecmp_pipe, creates the steering routes and writes the pipe configuration to the controller. Inputs without a default are required.

Production controller root: cloudspectra-relay-controller

The inputs that matter for a pipe:

Outputs include controller_url (http://<elastic-ip>:<listen_port>), elastic_ip, eip_allocation_id, table_name and security_group_id.

The cloudspectra provider

Both pipe resources live in the Cloud Spectra Gateway provider, cloudspectra/gateway. Only cloudspectra_pipe_config talks to anything; it needs two provider settings:

provider "cloudspectra" {
  controller_url = "http://203.0.113.10:8080"   # or CS_FLEET_CONTROLLER
  config_token   = var.fleet_config_token        # or CS_FLEET_CONFIG_TOKEN
}

Resource cloudspectra_ecmp_pipe

The pipe's intent, and the product's own derivation of it. It runs the same sizing and validation rules the relays are built from, at plan time, and creates no infrastructure: its delete is empty and it calls no cloud API. The deployment roots use it to size both fleets; on its own it is a sizing calculator (see Size a pipe before you build it).

Resource cloudspectra_pipe_config

The pipe configuration stored on the controller. The controller builds each relay's roster from it -- which sides exist and what is reachable through each -- and refuses to serve any roster without it, so the deployment roots write it for you (write_pipe_config, default true). The resource talks to the controller named by the provider's controller_url with its config_token.

Computed: revision (the compare-and-swap token), set_at, mode (always cloud_connector). Every write reads the stored revision first and names it; if someone changed the config in between, the controller answers HTTP 409 naming the other writer instead of overwriting. Destroying this resource only removes it from state: the controller has no delete verb, and removing a config under running relays would take the roster down with it.

Routing traffic into the pipe

Each side advertises its own ranges, and each fleet steers the other side's ranges into the pipe. The pipe roots wire that crossover once, from aws_advertise_cidrs and gcp_advertise_cidrs; everything else in your networks keeps its normal routing. A relay checks the ranges it was deployed with at start-up and refuses to start, with the reason in its log, unless they are IPv4, at most 8 per side, not 0.0.0.0/0, non-overlapping between the sides, and clear of 169.254.0.0/16 and 168.63.129.16.

AWS

• Target the zone's GWLB endpoint. In the route table of each workload subnet, route every Google Cloud range to the GWLB endpoint in the subnet's own zone (aws_endpoints_by_zone). A route cannot target the load balancer or the endpoint service. The pipe root can create these routes for you: list the route tables in aws_zone_route_tables.
• Never steer the relays' own subnets. The relay subnets' route table must not send the far side's ranges back to the endpoint.
• Admit the far side. Packets arrive with their original Google Cloud sources, so workload security groups and network ACLs must allow those ranges. Allow ICMP from the relays too: path-MTU messages come from them.
• Manage a table's routes one way. If you manage route tables with Terraform, do not mix inline route blocks with separate aws_route resources on the same table: Terraform then deletes the separate routes on a later apply, and the pipe silently stops carrying.

Google Cloud

• Tag the workloads. The pipe root creates one route per AWS range, next hop the internal load balancer, priority 900, applying only to VMs with a tag in gcp_client_tags (the evaluation root uses <name>-client). Routes in Google Cloud apply network-wide unless tagged, which is why the relays' tag (<name>-gcp-relay) must never be a client tag: a relay would become the next hop for traffic it is forwarding.
• Admit both directions. The relays' firewall rules admit the WireGuard band, Google's health checkers, ICMP and the AWS ranges. Add a rule letting your workload ranges reach the relays (target tag <name>-gcp-relay) and a rule letting the AWS ranges reach your workloads.

Which side opens connections

Connections opened from AWS workloads to Google Cloud workloads, and their replies, are the direction Cloud Spectra's live tests carry. On the AWS end the Gateway Load Balancer is used for ingress only: packets arriving from Google Cloud are delivered straight to the AWS host, so the load balancer never sees the first half of a connection opened from Google Cloud -- and a Gateway Load Balancer does not forward the reply half of a flow whose first half it never saw. Validate connections opened from the Google Cloud side before you depend on them.

Sizing and scaling

Sizing

Each relay is sized as 3 Gbps of pipe. target_gbps becomes a per-end relay count: ceil(target_gbps / 3), at least 1. That count is a target:

• the AWS end runs ceil(count / zones) relays in every zone, so it can run more than the target;
• the Google Cloud end runs exactly the target, spread over its zones.

The outputs report both: the target (relay_target_per_end or fleet_size) and what each end runs.

Capacity scales across many conversations, not within one. Every flow between one pair of addresses enters the pipe through a single relay, so one bulk transfer between two hosts gets one relay's worth of throughput however large the fleet. Many hosts talking across the pipe spread over all relays. And because each relay bonds with every relay on the far side, grow both ends together: growing one end alone adds little.

Size a pipe before you build it

The plan-only/ directory beside the cloudconnector example applies only cloudspectra_ecmp_pipe. It needs the cloudspectra/gateway provider and nothing else -- no cloud credentials -- and it creates nothing, so you can try shapes freely:

cd examples/cloudconnector/plan-only
terraform init
terraform apply \
  -var aws_vpc_id=vpc-0123456789abcdef0 \
  -var aws_subnet_id=subnet-0123456789abcdef0 \
  -var gcp_network=projects/my-project/global/networks/my-net \
  -var gcp_subnetwork=projects/my-project/regions/us-central1/subnetworks/my-subnet \
  -var target_gbps=10
terraform output plan_summary

With the defaults (ingress = "managed_lb", direct addressing on both sides) and target_gbps = 10, the summary reads:

mode cloud_connector: ECMP L3 forwarding across the fleet, source preserved
  servers:      4 (managed_lb ingress, direct addressing)
  client:       direct addressing, runs the CloudSpectra datapath
  source:       preserved end to end (no NAT)
  licence needs: tunnel-bond
  controller:   a peer-map container is stood up in your account
  billed as:    cloud_connector
  note: this mode stands up the peer-map controller container in your own account (roughly $15-50/mo); your client and peer data never leaves it

The same notes are printed as About this pipe warnings during terraform plan of the full deployment. A pipe sized to a single relay per end adds a note that it has no redundancy: losing that relay loses the pipe. The cost in the controller note is the derivation's rough estimate; what you pay for the controller is its AWS resources (see Cost considerations).

The bond ceiling

A relay's bond has one member per far-side relay, and member m listens on UDP wg_port + m. The firewalls on both ends open tunnel_members ports from wg_port (the wg_port_range output), so tunnel_members must cover the most relays either end will ever run. The plan warns when it is below the derived relay count or below what the AWS groups may reach (aws_per_zone_max times the number of zones). It can be 2 to 32; a relay bonds with at most 32 far relays in any case, and beyond that the highest-numbered far relays carry no traffic.

Changing the size

After the first apply the group sizes belong to the running system: Terraform ignores later changes to the AWS groups' desired capacity and to the Google Cloud group's target size, so that an apply never reverts a fleet that has scaled. Raising target_gbps raises the AWS groups' minimum size, which pulls them up; the Google Cloud group keeps its size until you resize it directly. Otherwise resize the groups directly or let the controller do it.

When a side changes size, the other side follows on its own. A new relay registers and the far side widens its bonds; a removed relay leaves the roster once its 90-second registration lapses, and the far side narrows. Each far relay rebuilds its bond with a brief restart of its datapath. In Cloud Spectra's measurements the far side widened about 2.5 minutes after a relay was added and narrowed about 7 minutes after one was removed; once the bond is rebuilt, traffic flows again within seconds.

Scaling by the controller

The controller can resize relay groups in one cloud (scaler_cloud), either on request (scaling = "manual") or from the relays' reported load (scaling = "auto"). It is off by default and acts only on the groups listed in scalable_groups. In automatic mode it grows a group above 70% utilisation and shrinks it below 30% (autoscale_high, autoscale_low), waiting 2 minutes before growing and 10 before shrinking, with a 5-minute cooldown, within autoscale_min and autoscale_max. Manual and automatic are exclusive. Keep tunnel_members above the largest size you allow.

High availability and failure behaviour

Monitoring and alerts

Pipe status

The controller derives the pipe's health each time it is asked:

curl -s -H "Authorization: Bearer <read-token>" \
  "http://<controller>:8080/fleet/v1/pipe/status?fleet=<fleet-id>"

A relay is idle when it is live but has carried nothing for 270 seconds, so a pipe with no traffic reads degraded. Two more read-only endpoints use the same token: GET /fleet/v1/pipe?fleet=<fleet-id> returns the stored configuration and its revision, and GET /fleet/v1/relays?fleet=<fleet-id> returns the relay registry.

Alerts

The controller turns health changes into alerts: pipe_down (critical), pipe_degraded, pipe_idle_relay, pipe_unplaced_relay, pipe_not_visible (warnings) and pipe_recovered. Every alert is logged by the controller, and POSTed as JSON (kind, title, message, severity, fleet_id, at) to pipe_alert_webhook when it is set. Alerts are edge-triggered with a six-hour cooldown per kind, and they are evaluated when the status endpoint is read -- so poll /fleet/v1/pipe/status from your monitoring, every minute or so, to get them.

On a relay

• Logs: journalctl -u cloudspectra-relay for the datapath agent; cloudspectra-relay-dash and cloudspectra-relay-upgrade are the dashboard and boot-time upgrade units. Look for relay: cloud connector pipe configured (it also names any far relays beyond the 32-member limit), relay: bond shape resolved and the line reporting the prefixes armed.
• Datapath facts: /run/cloudspectra/relay-facts.json records what the datapath was loaded with, including cs_pipe_nlocal, cs_pipe_nremote, cs_pipe_mtu, cs_wg_nmembers and cs_wg_nslots.
• Cloud health: the GWLB target group's health and the managed instance group's health show whether each relay's datapath is answering.

The relay's own dashboard (HTTPS on port 28300) has no pipe view; pipe health lives on the controller.

Upgrades

A relay runs the software in the image it booted from, and can move to a newer package at boot:

• Images follow release. The deployment root looks up the AWS AMI and the Google Cloud image for the release you name. Changing it updates the AWS launch template and the Google Cloud instance template, but running relays are not replaced: the Auto Scaling groups launch from the template's $Latest version with no instance refresh, and the managed instance group's update policy is OPPORTUNISTIC. New relays -- scale-out or replacement -- boot the new image; existing ones keep the old one until you replace them.
• In-place package upgrade at boot. Before it brings up its datapath, every relay runs a one-shot upgrade unit (cloudspectra-relay-upgrade.service). When a target version is set (package_version on the production pipe root), the relay downloads that exact package, verifies its checksum and signature, installs it and restarts into it. A failed upgrade is not fatal: the relay comes up on the software it already has. In-place upgrades never cross a major version.

To roll a fleet onto a new release without taking the pipe down, replace relays a few at a time on each side, keeping at least one relay in service per side and waiting for each replacement to register before the next. The far side picks up a new relay within a few minutes and drops a removed one after the registry's 90-second time-to-live (see Scaling):

# AWS: one Auto Scaling group per zone, named <name>-aws-<zone>
aws autoscaling start-instance-refresh --auto-scaling-group-name <name>-aws-us-east-1a

# Google Cloud: the regional managed instance group, named <name>-gcp-mig
gcloud compute instance-groups managed rolling-action replace <name>-gcp-mig \
  --region us-central1 --max-surge 1 --max-unavailable 0

Security model

Data in transit

Between the clouds, traffic travels only inside WireGuard tunnels between your own relays (kernel WireGuard; no pre-shared key is configured). Cloud Spectra is not on the data path. Inside each cloud, packets move between your hosts, the load balancers and the relays unencapsulated, as in any routed network. The datapath caps the TTL of forwarded packets at 16, so a routing loop dies quickly, and refuses compressed-protocol frames (IP protocol 253) arriving from outside the pipe.

Keys

Each relay generates its WireGuard key pair on first boot and stores the private key on the instance (/var/lib/cloudspectra/relay-self.key, mode 0600). Only the public key is sent to the controller and served to the far side, so no relay private key exists in Terraform state. A replacement instance gets a new key. Relay keys are not rotated automatically in this release.

Credentials

The evaluation root generates all of these in Terraform state and exposes them as a sensitive output. In production, generate them yourself, keep them in your secret store, and keep the state file encrypted and access-controlled.

The controller

The controller serves plain HTTP on listen_port (8080) at its Elastic IP; the deployment roots configure no TLS certificate for it, so the bearer tokens above cross the network unencrypted. Limit relay_cidrs to the addresses your relays and monitoring reach it from, and treat the read and write tokens accordingly. The controller instance requires IMDSv2.

The relays

• The WireGuard band (UDP from 28302) is open to any IPv4 source on both ends; packets that do not authenticate as a known far relay are dropped by WireGuard.
• AWS relays accept GENEVE (UDP 6081) and the health check only from the VPC's own range, require IMDSv2, and have encrypted root volumes.
• The relay datapath runs under systemd with only the capabilities it needs (CAP_NET_ADMIN, CAP_BPF, CAP_SYS_ADMIN, CAP_NET_RAW) and a read-only system. Its eBPF datapath is fetched signed at boot.

What source preservation means for your rules

Nothing on the pipe hides the sender. The far side's firewalls, security groups and logs see every host's own private address -- which is the point, and also means each side must explicitly admit the other side's ranges, at whatever granularity you choose.

Troubleshooting

Terraform's check warnings appear during plan and apply but do not stop them, so read them first. On a relay, start with journalctl -u cloudspectra-relay; on the controller, start with the pipe status.

Teardown

• Remove what you added. Delete your own routes to the GWLB endpoints, firewall rules and test instances first; Terraform cannot delete networks that still hold resources it does not manage.
• Production order. Destroy the pipe root before the controller root. Removing cloudspectra_pipe_config only removes it from Terraform state -- the controller has no delete for it, so the stored configuration stays until the controller itself is destroyed, and a later apply adopts it.
• The controller's durable state. With protect_durable_state = true (the default) the Elastic IP cannot be destroyed and the DynamoDB table has deletion protection, and the setting cannot be switched off on a live deployment. To destroy a protected controller, remove the address from Terraform state, disable the table's deletion protection, destroy, and then release the address yourself -- the commands are in the Quick Start's teardown step.
• Billing. Destroying infrastructure does not cancel your Cloud Spectra subscription.

Cost considerations

• Licence. Per relay vCPU, both ends, hourly; the second vCPU of each instance is free (see Pricing). The controller is not licensed.
• Inter-cloud data. Traffic between the clouds crosses the internet, so each cloud bills the bytes leaving it as internet egress, in both directions. This is usually the largest cost of a busy pipe; where a private interconnect with cheaper transfer already exists, compare.
• AWS. The relay instances, the Gateway Load Balancer and its endpoints, the controller instance, its Elastic IP and its on-demand DynamoDB table. Cross-zone load balancing is off and each zone's subnets should route to their own zone's endpoint, so no inter-zone transfer is added.
• Google Cloud. The relay VMs, the internal load balancer, and Cloud NAT where the root creates it.
• Public addresses. Both clouds bill public IPv4 addresses per hour; that is why Google Cloud relays default to IPv6.
• Right-sizing. Relays are sized at 3 Gbps each; size by target_gbps rather than by instance size alone, and remember one conversation never uses more than one relay.