CloudVpc -- User Guide v1.0.0

This guide is the operating reference for CloudVpc: how it is built, how to configure each supported topology, and how to scale, upgrade, monitor, secure, troubleshoot and remove it. It assumes you have read the Overview; the Quick Start walks through one deployment end to end.

Architecture

Components

Data path

  VPC client or load balancer                        worker (OCI or DigitalOcean)
            |                                                     ^
            v  to 10.43.0.21 (the relay's private IPv4)           |  cswg0 wears 10.43.0.21
   relay --- eBPF redirect into the WireGuard tunnel -------------+
            (no address translation)

  worker --- 169.254.169.254 (metadata) --- tunnel --- relay --- VPC metadata service
  worker --- VPC range -------------------- tunnel --- relay --- VPC
  worker --- other destinations ----------- the worker's own cloud (split tunnel)
  worker --- another worker --------------- direct, GENEVE over UDP 6081 (fallback: via the relays)

• North-south. Traffic addressed to the relay's private IPv4 is redirected into the tunnel by eBPF programs on the relay and delivered to the worker, which holds that address on its tunnel interface and answers as it.
• Metadata. The worker routes 169.254.169.254/32 through the tunnel. The relay forwards without consuming a hop and caps the metadata service's replies to one hop, so a process on the worker host gets an answer and a container behind it does not.
• Split tunnel. The tunnel carries the VPC's range and the metadata address. The ranges are issued by the relay when the worker enrols and refreshed afterwards. The relay never issues a default route; a full tunnel is a per-client choice that the CloudVpc configurations do not make.
• East-west. Workers that know each other's host address exchange traffic directly, in GENEVE over UDP port 6081, spread across four source ports (49152-49155). Until a peer's address is known, traffic goes through the relays.

What happens at boot

A relay boots the Cloud Spectra Gateway image with its configuration in user data, and first upgrades itself in place if the image is older than package_version. When it has a rendezvous block, it claims an address from its own Availability Zone's block before it serves anything; if it cannot hold one, it refuses to start, its health check fails, and the Auto Scaling group replaces it. A running relay re-asserts its claim every 3 minutes. The relay fetches its datapath from delivery_url and proves its licence; a relay without a valid licence keeps running but is parked and forwards nothing.

A worker runs a short cloud-init document: it writes /etc/cloudspectra/licence and /etc/cloudspectra/worker.yaml (both mode 0600), downloads the downloader, installs the client, and runs cloudspectra_relay_attach bootstrap-worker. That command executes an ordered plan and stops at the first step that fails: wait for the distribution's package jobs, install base packages, install the operator SSH key, write the client environment, pair with a relay and bring the tunnel up, wait until the VPC metadata service answers through the tunnel, and then -- depending on the configuration -- join the Kubernetes cluster or start the demo application on port 8080. It narrates each step to the serial console and to /var/log/cloudspectra-worker-bootstrap.log. bootstrap-worker -dry-run prints every step, file and command without changing anything.

Addressing

Each relay announces which of its addresses does what:

In the autoscaling topologies the identity is disposable: AWS assigns each relay's private IPv4 at launch, so a replacement relay has a new one. What stays stable is the load balancer and the rendezvous block.

The rendezvous block

Each relay subnet has its own IPv6 /64, and Terraform pre-allocates addresses in it: slots_per_az per Availability Zone in the pool configurations (default 20, at most 200), and 8 in the EKS configuration. A relay claims an address with the EC2 AssignIpv6Addresses call. AWS assigns an IPv6 address to exactly one network interface, so a second relay asking for the same address is refused and tries the next. Nothing else records who holds what; run the watch_claims output to see the current claims. AWS releases an address when the relay's interface is deleted.

Enrolment and pairing

• The fleet enrolment token. Terraform generates one token per fleet and seeds a copy into every relay at its first boot. In the pool configurations each relay accepts it once, within enrol_token_ttl (default 168h) of its own boot: the first worker to reach a relay pairs with it, and the next is refused and moves on. That is how N workers spread over N relays. A replacement relay seeds a fresh copy. The token authorises enrolment and a read-only download, nothing else.
• Keys stay on the machine that uses them. A worker generates its own WireGuard key pair and sends only the public half when it enrols. The relays in these configurations also generate their own keys, so no WireGuard private key is written to Terraform state.
• Workers spread their attempts. Each worker walks the rendezvous list from a starting point derived from its host name, so workers booting together do not all try the same relay first.

How the number of workers is kept equal to the number of relays depends on the topology:

On scale-in, the relay group's lifecycle hook (notify-worker-on-terminate) holds a terminating relay so that its worker can be withdrawn first -- for up to terminate_hook_seconds (default 120) in the pool configurations. The hook is best effort and lets the termination continue on timeout; the worker's own check, or the controller's next pass, is what guarantees that no worker is left orphaned.

Topologies

These docs cover three Terraform configurations. Each creates its own dual-stack AWS VPC for the relays.

OCI pool -- aws-asg-oci-pool

The topology of the Quick Start. Terraform creates the dual-stack VPC (one public subnet and one IPv6 /64 per Availability Zone, in at least two zones), the relay Auto Scaling group behind an Application Load Balancer, the peer map table, an IAM user for the workers, and an OCI VCN with IPv6 holding an instance pool of workers. Workers use the latest Canonical Ubuntu 24.04 image for the chosen shape. After creation Terraform ignores the pool's size; the workers own it.

• The workers resize their pool with OCI instance-principal credentials, which needs a tenancy-level dynamic group and policy. The configuration creates them unless manage_oci_iam = false (see below).
• Each worker holds the access key of an IAM user named <name>-client-reader, which can call autoscaling:DescribeAutoScalingGroups and nothing else, to read the relay count.
• There is no fleet-wide safety rail: each worker acts on what it sees. The rules above -- act only on positive readings, never go below the floor, size from the group rather than from probes -- are what keep that safe.

DigitalOcean pool -- aws-asg-do-pool

The same AWS half as the OCI pool; the workers are a fixed number of DigitalOcean Droplets in a DigitalOcean VPC. It takes the same inputs as the OCI pool, without the oci_*, worker_floor and worker_initial_size settings, and adds:

EKS -- eks-oci-asg

One Amazon EKS cluster. Its relays are an Auto Scaling group that is a self-managed node group, and its workers run in OCI or DigitalOcean (worker_cloud = "oci" or "do") and join the cluster as nodes, each named after the relay whose identity it wears and labelled cloudspectra.ai/compute-location. The configuration also creates a small native node group (native_count, default 1) and installs Calico in VXLAN mode as the CNI (calico_version, default v3.29.1; pod_cidr, default 192.168.128.0/17).

Nothing is paired at plan time. With in_cluster_controller = true, the controller runs as the cloudspectra-master Deployment in the cloudspectra-system namespace; the relays reach it on a NodePort inside the VPC, and it has no public endpoint. Exactly one of in_cluster_controller or master_endpoint (an external controller) must be set; the plan is refused otherwise.

On the machine that runs Terraform you also need Python 3, the AWS CLI and kubectl, which the configuration uses to bootstrap the cluster, and the Python boto3 package -- plus the oci Python SDK for OCI workers -- which it uses to drain the fleet before a destroy. Useful outputs: kubeconfig_command, verify_nodes, scale_command, relay_asg, fleet_id, app_url.

eval "$(terraform output -raw kubeconfig_command)"
eval "$(terraform output -raw verify_nodes)"

NAME                         STATUS  VERSION  COMPUTE-LOCATION  INSTANCE-TYPE
ip-10-42-0-31.ec2.internal   Ready   v1.32.x  oci               VM.Standard.A1.Flex
ip-10-42-0-87.ec2.internal   Ready   v1.32.x  oci               VM.Standard.A1.Flex

Configuration reference

Each configuration ships a generated terraform.tfvars.example listing every variable with its default, required ones first. The tables below cover the pool configurations; aws-asg-do-pool differs as listed above, and the EKS configuration's main inputs are listed above.

Required

Optional

Outputs

OCI IAM without tenancy rights

The OCI pool's workers resize their own pool with instance-principal credentials, which needs a dynamic group and a policy at tenancy level. OCI accepts these only from the tenancy's home region, and only from a user who may change tenancy IAM; otherwise the apply fails (for example with 403-NotAllowed and a request to use the home region). In that case set manage_oci_iam = false and have a tenancy administrator create the equivalents:

dynamic group  ALL {instance.compartment.id = '<compartment OCID>'}
policy         Allow dynamic-group <name> to use instance-pools in compartment id <compartment OCID>
               Allow dynamic-group <name> to inspect instance-pools in compartment id <compartment OCID>

Without them the fleet pairs correctly, but the pool never follows the relay group.

The cloudspectra/gateway Terraform provider

The public provider's cloudspectra_ecmp_pipe resource accepts mode = "cloud_vpc" and reports at plan time what that mode derives to -- for example, that the client end runs the Cloud Spectra datapath and how the pipe is billed. It is a planning check only: it creates no relays, workers or networks. CloudVpc itself is deployed with the configurations above.

Scaling

The relay Auto Scaling group is the only place the fleet size is decided. Resize it with the scale output's command, with terraform apply -var relay_desired=N, or with a scaling policy.

• OCI pool: the workers follow within a poll interval.
• DigitalOcean pool: set worker_count to match and apply.
• EKS: the controller follows within a reconcile interval.

Scaling on pair load

A relay's EC2 metrics describe the relay, not the worker running the workload. So, in the pool configurations, each relay publishes its pair's load to CloudWatch every 30 seconds, and pair_scaling_enabled = true creates:

• a target-tracking policy on PairCPUPercent -- the higher of the relay's and the worker's CPU, never their average -- that holds it at pair_cpu_target (default 70%);
• an alarm on the relays' packets-per-second and bandwidth-out allowance-exceeded rates that adds one relay when either has been above zero for two consecutive minutes.

A pair publishes only while its worker is reporting: a relay without a fresh worker reading withholds the pair metrics instead of publishing half of them. Leave pair scaling off wherever something else sets the group's size; the EKS configuration does not offer it.

Licence limits that affect scaling

• Developer lane: an account may run at most 16 vCPUs. Relays beyond the limit park and carry no traffic until room frees -- for example, eight 2-vCPU relays fit.
• Enterprise lane: no account limit, but a relay with fewer than 4 vCPUs is refused.

Upgrades

Every configuration pins an exact package_version; nothing follows a "latest" pointer.

• Relays read the version they were given at first boot and, at every boot, upgrade themselves in place to it from the release host when the image is older. The download is checked against its SHA-256 and the release signature. Upgrades across a major version are refused in place; use a new relay image (relay_ami_id) for those.
• Workers install the client version given by package_version once, at first boot. There is no in-place client upgrade.

To move a fleet to a new release, change package_version (and relay_ami_id if the image changes) and apply, then replace instances so they boot with the new configuration. Running relays keep what they booted with: replace them one at a time -- for example by terminating a relay and letting the group launch its successor. In the OCI pool and the EKS topology, a replaced relay's worker is retired and its successor is launched with the new client. In the DigitalOcean pool the apply itself replaces the Droplets, because their user data changes.

Monitoring

Logs

On the relay, wg show shows nothing: the relay runs WireGuard inside its eBPF datapath rather than as a kernel interface. On a worker, the tunnel interface is cswg0.

Relay dashboard

Each relay serves a small HTTPS dashboard on port 28300 with a self-signed certificate. In the pool configurations that port is open to the internet over IPv6 only, so reach it at the relay's rendezvous address. The login password is generated at first boot into /var/lib/cloudspectra/credentials on the relay; read it over SSH with sudo cat. The Monitoring tab shows the relay's CPU, memory, storage, network throughput and egress allowance.

Security model

• Accounts are the boundary. Relays, their IAM role and the peer map live in your AWS account; workers live in your OCI or DigitalOcean account. Tags select resources; they do not grant access.
• The worker holds the relay's IAM role. Any process on the worker host that can reach the metadata service obtains temporary credentials for the relay's instance profile -- the role is <name>-relay in the pool configurations -- including the permissions CloudVpc itself grants the relay. Add your workload's permissions to that role, and nothing more.
• Containers do not. With the default imds_hop_limit = 1, and with the relay capping metadata replies to one hop, a container one forwarding hop away from the worker host cannot reach the metadata service. The relays require IMDSv2.
• Encrypted transport. All relay-worker traffic is inside WireGuard. Workers and relays generate their own keys, and no WireGuard private key is stored in Terraform state. Direct worker-to-worker GENEVE traffic is not encrypted by CloudVpc; it stays between the workers on their own cloud network, and the OCI configuration allows it only from the worker subnet.
• Narrow enrolment. The fleet's enrolment token authorises only enrolment and a read-only download; changes in the dashboard need a signed-in session. In the pool configurations each relay accepts the token once, within enrol_token_ttl of its boot.
• Exposed ports. Each relay accepts the WireGuard port (UDP 28302) and the enrolment and health port (TCP 28300) from anywhere, because workers in another cloud cannot be known by address in advance. Recovery SSH (22 and 28301) is limited to admin_cidrs, empty by default. There is no IPv6 NAT: a relay is reachable at its rendezvous address as soon as it claims it, and the security group is what limits that.
• Worker firewalls. In the OCI pool the workers' security list admits only SSH from admin_cidrs and GENEVE (UDP 6081) from the worker subnet; application traffic arrives through the tunnel the worker opened. The DigitalOcean pool configuration creates no DigitalOcean firewall; add one.
• Secrets on the worker. The licence key (/etc/cloudspectra/licence) and worker.yaml -- which holds the enrolment token and, in the OCI pool, the read-only AWS key -- are written with mode 0600. They travel in the worker's launch metadata and are present in Terraform state; protect the state accordingly.

Access for debugging

The relays and workers run no SSM agent, so SSH is the way in. It is closed unless you open it.

• Relays. Supply your SSH public key (oci_ssh_public_key, or ssh_public_key_override on DigitalOcean; ~/.ssh/id_rsa.pub by default) or an EC2 key pair (relay_key_name), and set admin_cidrs. Log in as admin: on port 22 while the relay boots, and on port 28301 once it is running, because the relay moves SSH there.
• Workers. The same key is installed for the workers' login users. Log in as ubuntu on OCI and as root on DigitalOcean.

Troubleshooting

When you contact support@cloudspectra.ai, include the configuration name, its terraform output (without secrets), the relay journal and the worker's bootstrap log.

Teardown

Run terraform destroy in the configuration you applied, with the same variables and credentials.

• Pool configurations. Everything is in Terraform state, including the OCI pool and the DigitalOcean Droplets. If the destroy stops on a DependencyViolation about mapped public addresses, run it again after a minute.
• EKS. The workers are launched by the controller, not by Terraform, so the configuration drains the fleet first: a destroy-time step scales the relay group to zero, releases the relays' lifecycle hooks, waits for the controller to retire the workers (drain_timeout_seconds, default 300), terminates any that remain, and only then lets the cluster, the relays and the worker network go. It needs Python 3 with boto3, the oci SDK for OCI workers or DIGITALOCEAN_TOKEN for DigitalOcean workers, and cloudspectra_fleet_manifest. If the drain cannot finish, the destroy still runs and reports what it could not delete.
• Afterwards, confirm in each cloud that no instance tagged with your fleet remains. Destroying the deployment does not change your Cloud Spectra subscription; manage it on your account page.