1. Introduction
Private Service Connect is a capability of Google Cloud networking that allows consumers to access producer services. This includes the ability to connect to Google APIs via a private endpoint that is hosted within the user's(typically consumer) VPC.
Additionally, PSC Backends can be used in conjunction with Google Cloud proxy load balancers to point to region specific Google APIs. Using PSC Backends provides more granular consumer side controls such as:
- Choosing which Google API services are available using a URL Map
- Deeper observability
- Cloud Armor Integration
- Custom URLs
- Advanced Traffic Management
The list of available services and their regional APIs can be found here.
In this codelab, you'll create a PSC Backend pointing to the regional Cloud KMS API and test connectivity to that API.
What you'll learn
- Create a Cloud Key Management Service (KMS) key ring and key.
- Create an Internal Application Load Balancer with a PSC Backend pointing to a Cloud KMS regional API
- Creating a Cloud DNS Managed Private Zone and A Record.
- Access the regional Cloud KMS
What you'll need
- A Google Cloud Project with ideally "Owner" or full "Editor" permissions.
2. Codelab Topology
A consumer VPC will be created with one subnet in the europe-west9 region for hosting a VM, the Internal Regional Application Load Balancer forwarding rule, and the PSC Backend and one proxy only subnet for use with the load balancer. We will create a Key Ring and Key in Cloud Key Management System (KMS) in the europe-west region. We will then create the load balancer and PSC Backend to resolve to the regional KMS API in europe-west9.
3. Setup and Requirements
Self-paced environment setup
- Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.
- The Project name is the display name for this project's participants. It is a character string not used by Google APIs. You can always update it.
- The Project ID is unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference your Project ID (typically identified as
PROJECT_ID
). If you don't like the generated ID, you might generate another random one. Alternatively, you can try your own, and see if it's available. It can't be changed after this step and remains for the duration of the project. - For your information, there is a third value, a Project Number, which some APIs use. Learn more about all three of these values in the documentation.
- Next, you'll need to enable billing in the Cloud Console to use Cloud resources/APIs. Running through this codelab won't cost much, if anything at all. To shut down resources to avoid incurring billing beyond this tutorial, you can delete the resources you created or delete the project. New Google Cloud users are eligible for the $300 USD Free Trial program.
Start Cloud Shell
While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.
From the Google Cloud Console, click the Cloud Shell icon on the top right toolbar:
It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:
This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this codelab can be done within a browser. You do not need to install anything.
4. Before you begin
Enable APIs
Inside Cloud Shell, make sure that your project id is set up
gcloud config list project gcloud config set project <project-id> export PROJECT_ID=$(gcloud config get-value project) export REGION=europe-west9 export ZONE=europe-west9-a echo $PROJECT_ID echo $REGION echo $ZONE
Enable all necessary services
gcloud services enable compute.googleapis.com gcloud services enable servicedirectory.googleapis.com gcloud services enable dns.googleapis.com gcloud services enable cloudkms.googleapis.com
5. Create VPC network, Subnets, and Firewall Rules
Create VPC Network
From Cloud Shell
# Set environment variables export VPC_NAME="consumer-vpc" export SUBNET_NAME="consumer-subnet-1" # Create VPC network gcloud compute networks create ${VPC_NAME} \ --subnet-mode=custom \ --bgp-routing-mode=regional
Create Subnets
From Cloud Shell
gcloud compute networks subnets create ${SUBNET_NAME} \ --network=${VPC_NAME} \ --region=${REGION} \ --range=10.0.0.0/24 \ --enable-private-ip-google-access
In this lab, you will create an internal L7 regional load balancer to point to regional API backends. This load balancer is a proxy load balancer, therefore you need to create a "proxy subnet" that is dedicated to the load balancer to perform the proxy. More information about the proxy-only subnet can be found here.
From Cloud Shell
gcloud compute networks subnets create eu-west9-proxy-subnet \ --network=${VPC_NAME} \ --region=${REGION} \ --range=10.100.100.0/24 \ --purpose=REGIONAL_MANAGED_PROXY \ --role=ACTIVE
Create Firewall Rules
For this lab, you will be using IAP to connect to the instances you create. If you prefer not to use IAP, you can skip this step, and instead add public IP addresses on the instance and create a firewall rule that allows ingress on TCP port 22 from 0.0.0.0/0.
To allow IAP to connect to your VM instances, create a firewall rule that:
- Applies to all VM instances that you want to be accessible by using IAP.
- Allows ingress traffic from the IP range 35.235.240.0/20. This range contains all IP addresses that IAP uses for TCP forwarding.
From Cloud Shell
gcloud compute firewall-rules create allow-ssh-iap \ --network=${VPC_NAME} \ --allow tcp:22 \ --source-ranges=35.235.240.0/20
6. Create Cloud NAT
A Cloud NAT must be created to download Linux package distributions.
Create Cloud Router
From Cloud Shell
gcloud compute routers create crnat \ --network=${VPC_NAME} \ --region=${REGION}
Create Cloud NAT
From Cloud Shell
gcloud compute routers nats create europe-nat \ --router=crnat \ --auto-allocate-nat-external-ips \ --nat-all-subnet-ip-ranges \ --enable-logging \ --region=${REGION}
7. Create Key Management Key Ring and Key
From Cloud Shell
gcloud kms keyrings create europe-kr \ --location ${REGION}
From Cloud Shell
gcloud kms keys create europe-key \ --location ${REGION} \ --keyring europe-kr \ --purpose encryption
From Cloud Shell, confirm the key ring and key were created successfully in the europe-west region.
gcloud kms keys list \ --location ${REGION} \ --keyring europe-kr
EXPECTED RESULT
NAME: projects/<project-id>/locations/europe-west9/keyRings/europe-kr/cryptoKeys/europe-key PURPOSE: ENCRYPT_DECRYPT ALGORITHM: GOOGLE_SYMMETRIC_ENCRYPTION PROTECTION_LEVEL: SOFTWARE LABELS: PRIMARY_ID: 1 PRIMARY_STATE: ENABLED
Note the full path name given for the keys as you will be using this to connect to later.
8. Create Client VM and connect to KMS API
Next you'll create a client VM, SSH into the VM, and test the resolution of the global KMS API. This test represents the default option for resolution global Google APIs.
From Cloud Shell
#Create the startup script touch startup.sh #Open the startup.sh file using a text editor of your choice (e.g., nano, vim, gedit, etc.) nano startup.sh #Paste the following script content into the startup.sh file #! /bin/bash sudo apt-get update sudo apt-get install dnsutils -y sudo apt-get install tcpdump -y #Save the changes you made to the startup.sh file #Use the chmod command to make the script executable chmod +x startup.sh #Create the VM instance gcloud compute instances create client-vm \ --network="${VPC_NAME}" \ --subnet="${SUBNET_NAME}" \ --zone="europe-west9-a" \ --machine-type="e2-medium" \ --no-address \ --scopes="https://www.googleapis.com/auth/cloud-platform" \ --image-family="debian-12" \ --image-project="debian-cloud" \ --metadata-from-file="startup-script=./startup.sh"
Next, you will need to update the default Compute service account to gain access to the KMS key you created. The default compute service account will be in the format <Project_Number> -compute@developer.gserviceaccount.com. To get the Project Number run the following command from the Cloud Shell and copy the number on the last line of the return results.
gcloud projects describe $PROJECT_ID
Update the default Compute service account to gain access to the KMS key you created.
From Cloud Shell
gcloud kms keys add-iam-policy-binding europe-key \ --location $REGION \ --keyring europe-kr \ --member serviceAccount:<project_number>-compute@developer.gserviceaccount.com \ --role roles/cloudkms.admin
Create an additional cloud shell terminal by clicking + (screenshot below)
In tab 2, tunnel through IAP to SSH into client-vm. Note that environment variables will not carry and you'll need to add your project-id to the command below.
From Cloud Shell
# Set the environment variable export PROJECT_ID=$(gcloud config get-value project) # ssh into the client-vm gcloud beta compute ssh --zone europe-west9-a "client-vm" \ --tunnel-through-iap \ --project $PROJECT_ID
Connect to the global KMS API using the KMS key name you noted earlier.
From tab 2, client-vm
# Store the access token in a variable TOKEN=$(gcloud auth print-access-token) # Run the full command with maximum verbosity curl -v \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ 'https://cloudkms.googleapis.com/v1/projects/<project-id>/locations/europe-west9/keyRings/europe-kr/cryptoKeys/europe-key'
EXPECTED RESULT
* Trying 216.58.214.74:443... * Connected to cloudkms.googleapis.com (216.58.214.74) port 443 (#0) * ALPN: offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * CAfile: /etc/ssl/certs/ca-certificates.crt * CApath: /etc/ssl/certs * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN: server accepted h2 * Server certificate: * subject: CN=upload.video.google.com * start date: Aug 26 07:12:45 2024 GMT * expire date: Nov 18 07:12:44 2024 GMT * subjectAltName: host "cloudkms.googleapis.com" matched cert's "*.googleapis.com" * issuer: C=US; O=Google Trust Services; CN=WR2 * SSL certificate verify ok. * using HTTP/2 * h2h3 [:method: GET] * h2h3 [:path: /v1/projects/<project-id>/locations/europe-west9/keyRings/europe-kr/cryptoKeys/europe-key] * h2h3 [:scheme: https] * h2h3 [:authority: cloudkms.googleapis.com] * h2h3 [user-agent: curl/7.88.1] * h2h3 [accept: */*] * h2h3 [authorization: Bearer dsnkjfdnvjfd * h2h3 [content-type: application/json] * Using Stream ID: 1 (easy handle 0x55ed8bb7ece0) > GET /v1/projects/<project-id>/locations/europe-west9/keyRings/europe-kr/cryptoKeys/europe-key HTTP/2 > Host: cloudkms.googleapis.com > user-agent: curl/7.88.1 > accept: */* > authorization: Bearer dsnkjfdnvjfd > content-type: application/json > < HTTP/2 200 < content-type: application/json; charset=UTF-8 < vary: X-Origin < vary: Referer < vary: Origin,Accept-Encoding < date: Tue, 24 Sep 2024 18:25:42 GMT < server: ESF < cache-control: private < x-xss-protection: 0 < x-frame-options: SAMEORIGIN < x-content-type-options: nosniff < accept-ranges: none < { "name": "projects/<project-id>/locations/europe-west9/keyRings/europe-kr/cryptoKeys/europe-key", "primary": { "name": "projects/<project-id>/locations/europe-west9/keyRings/europe-kr/cryptoKeys/europe-key/cryptoKeyVersions/1", "state": "ENABLED", "createTime": "2024-09-24T17:56:01.905156045Z", "protectionLevel": "SOFTWARE", "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION", "generateTime": "2024-09-24T17:56:01.905156045Z" }, "purpose": "ENCRYPT_DECRYPT", "createTime": "2024-09-24T17:56:01.905156045Z", "versionTemplate": { "protectionLevel": "SOFTWARE", "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION" }, "destroyScheduledDuration": "2592000s" } * Connection #0 to host cloudkms.googleapis.com left intact
Check to see where DNS resolves the KMS endpoint.
From tab2, client-vm
dig cloudkms.googleapis.com
EXPECTED RESULT
<<>> DiG 9.18.28-1~deb12u2-Debian <<>> cloudkms.googleapis.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62617 ;; flags: qr rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;cloudkms.googleapis.com. IN A ;; ANSWER SECTION: cloudkms.googleapis.com. 300 IN A 142.250.74.234 cloudkms.googleapis.com. 300 IN A 142.250.75.234 cloudkms.googleapis.com. 300 IN A 216.58.214.170 cloudkms.googleapis.com. 300 IN A 172.217.20.170 cloudkms.googleapis.com. 300 IN A 172.217.20.202 cloudkms.googleapis.com. 300 IN A 216.58.215.42 cloudkms.googleapis.com. 300 IN A 216.58.213.74 cloudkms.googleapis.com. 300 IN A 142.250.179.74 cloudkms.googleapis.com. 300 IN A 142.250.179.106 cloudkms.googleapis.com. 300 IN A 142.250.178.138 cloudkms.googleapis.com. 300 IN A 142.250.201.170 cloudkms.googleapis.com. 300 IN A 172.217.18.202 cloudkms.googleapis.com. 300 IN A 216.58.214.74 ;; Query time: 4 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP) ;; WHEN: Wed Oct 23 19:58:58 UTC 2024 ;; MSG SIZE rcvd: 260
From tab2, client-vm
dig cloudkms.europe-west9.rep.googleapis.com
EXPECTED RESULT
<<>> DiG 9.18.28-1~deb12u2-Debian <<>> cloudkms.europe-west9.rep.googleapis.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2736 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;cloudkms.europe-west9.rep.googleapis.com. IN A ;; ANSWER SECTION: cloudkms.europe-west9.rep.googleapis.com. 3043 IN A 34.1.65.232 ;; Query time: 0 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP) ;; WHEN: Wed Oct 23 20:00:04 UTC 2024 ;; MSG SIZE rcvd: 85
The default behavior for Google APIs is to use the Global API service endpoint. This endpoint will resolve to a list of public IPs. The dig for cloudkms.googleapis.com shows this. (Note: The IP address you see may be a different external IP address. This is normal Google APIs behavior.)
By using Regional Google API endpoints with PSC, you are able to meet regional requirements for API traffic as well as change the resolution from public to private. The dig to cloudkms.europe-west9.rep.googleapis.com shows that at this point, the resolution to the regional KSM API endpoint is still public.
In the following sections we will migrate from using the Global KMS API endpoint to the regional endpoint and change the resolution to private using PSC Backends.
9. Create the PSC NEG and Load Balancer
For the next section, switch back to the first tab in Cloud Shell.
You will create an Internal HTTP(S) Load Balancer with a network endpoint group(NEG) pointing to the europe regional KMS endpoint as a backend service. The forwarding rule of the load balancer acts as the Private Service Connect(PSC) endpoint.
Create the Network Endpoint Group (NEG) with type Private Service Connect and target service europe-west9-cloudkms.example.com
From Cloud Shell
#Set the necessary variables NEG_NAME="l7psc-kms-neg" PSC_TARGET="cloudkms.europe-west9.rep.googleapis.com" #Create the Private Service Connect NEG gcloud compute network-endpoint-groups create ${NEG_NAME} \ --project=${PROJECT_ID} \ --region=${REGION} \ --network-endpoint-type=PRIVATE_SERVICE_CONNECT \ --psc-target-service=${PSC_TARGET} # Verify the NEG creation gcloud compute network-endpoint-groups describe ${NEG_NAME} \ --project=${PROJECT_ID} \ --region=${REGION}
Create the backend service for the load balancer.
From Cloud Shell
# 1. Set the necessary variables BACKEND_SERVICE_NAME="l7-psc-kms" # 2. Create the backend service gcloud compute backend-services create $BACKEND_SERVICE_NAME \ --load-balancing-scheme=INTERNAL_MANAGED \ --protocol=HTTPS \ --region=$REGION \
Add the NEG to the backend service.
From Cloud Shell
gcloud compute backend-services add-backend $BACKEND_SERVICE_NAME \ --network-endpoint-group=${NEG_NAME} \ --region=$REGION
Create the URL map for the load balancer.
From Cloud Shell
gcloud compute url-maps create l7-psc-url-map \ --default-service=l7-psc-kms \ --region=$REGION
Create the Path Matcher for the Custom URL that the endpoint will use.
From Cloud Shell
gcloud compute url-maps add-path-matcher l7-psc-url-map \ --path-matcher-name=example \ --default-service=l7-psc-kms \ --region=$REGION
Create the host rule for the custom url europe-west9-cloudkms.example.com.
From Cloud Shell
gcloud compute url-maps add-host-rule l7-psc-url-map \ --hosts=europe-west9-cloudkms.example.com \ --path-matcher-name=example \ --region=$REGION
Create the target HTTPS proxy for the load balancer. This requires creating a regional SSL certificate resource. Please see steps to create a self-signed certificate here. We'll create a self-signed certificate using openssl and use that to create a regional certificate resource on GCP. The target https proxy will use this certificate.
From Cloud Shell
# Set environment variables export CERT_NAME="my-ssl-cert" # Generate a private key openssl genrsa -out private.key 2048 # Create a configuration file for the CSR cat > csr_config.cnf << EOF [req] default_bits = 2048 prompt = no default_md = sha256 req_extensions = req_ext distinguished_name = dn [dn] CN = example.com O = My Organization C = US [req_ext] subjectAltName = @alt_names [alt_names] DNS.1 = example.com DNS.2 = *.example.com EOF # Create a CSR using the configuration openssl req -new -key private.key -out server.csr -config csr_config.cnf # Create a self-signed certificate using the CSR openssl x509 -req -days 365 -in server.csr -signkey private.key -out server.crt \ -extfile csr_config.cnf -extensions req_ext # Verify the certificate openssl x509 -in server.crt -text -noout #Create a regional SSL certificate resource gcloud compute ssl-certificates create ${CERT_NAME} \ --region=${REGION} \ --certificate=server.crt \ --private-key=private.key #Create the target HTTPS proxy for the load balancer gcloud compute target-https-proxies create psc-http-proxy \ --region=${REGION} \ --url-map=l7-psc-url-map \ --ssl-certificates=${CERT_NAME}
Create the forwarding rule for the load balancer which will act as the Private Service Connect Endpoint. The IP address for the forwarding rule must belong to the a regional subnet in the VPC that is in the same region of the API endpoint you are accessing.
From Cloud Shell
# Set environment variables export PROXY_NAME="psc-http-proxy" # Create the forwarding rule gcloud compute forwarding-rules create kms-lb-rule \ --project=${PROJECT_ID} \ --region=${REGION} \ --load-balancing-scheme=INTERNAL_MANAGED \ --network=${VPC_NAME} \ --subnet=${SUBNET_NAME} \ --target-https-proxy=${PROXY_NAME} \ --target-https-proxy-region=${REGION} \ --address=10.0.0.100 \ --ports=443
10. DNS Configuration
In this section you will create a private DNS zone for example.com, and an A record pointing to the forwarding rule we created in the last step.
Create Managed DNS Private Zone.
From Cloud Shell
# Set environment variables export LB_RULE_NAME="kms-lb-rule" export DNS_ZONE_NAME="example-com-private-zone" # Create the private DNS zone gcloud dns managed-zones create ${DNS_ZONE_NAME} \ --dns-name="example.com." \ --description="Private DNS zone for example.com" \ --visibility=private \ --networks=${VPC_NAME}
Create A record for europe-west9-cloudkms.example.com.
From Cloud Shell
gcloud dns record-sets transaction start \ --zone=${DNS_ZONE_NAME} gcloud dns record-sets transaction add 10.0.0.100 \ --name=europe-west9-cloudkms.example.com \ --ttl=300 \ --type=A \ --zone=${DNS_ZONE_NAME} gcloud dns record-sets transaction execute \ --zone=${DNS_ZONE_NAME}
11. Connect to Regional KMS API through PSC
Switch back to client-vm in tab 2 to run tcpdump to see all connection details and test the connections to the regional KMS endpoint through the PSC Backend.
sudo tcpdump -i any net 10.0.0.100 or port 53 -n
Open a 3rd tab in Cloud Shell and SSH into client-vm.
# Set environment variables export PROJECT_ID=$(gcloud config get-value project) export KMS_HOSTNAME="europe-west9-cloudkms.example.com" export KEY_RING="europe-kr" export KEY_NAME="europe-key" export REGION="europe-west9" # Command to access the specific key curl -k "https://${KMS_HOSTNAME}/v1/projects/${PROJECT_ID}/locations/${REGION}/keyRings/${KEY_RING}/cryptoKeys/${KEY_NAME}" \ -H "Authorization: Bearer $(gcloud auth print-access-token)"
EXPECTED RESULT for the curl command + TCPDUMP
{ "name": "projects/<project-id>/locations/europe-west9/keyRings/europe-kr/cryptoKeys/europe-key", "primary": { "name": "projects/<project-id>/locations/europe-west9/keyRings/europe-kr/cryptoKeys/europe-key/cryptoKeyVersions/1", "state": "ENABLED", "createTime": "2024-10-10T18:50:24.357027036Z", "protectionLevel": "SOFTWARE", "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION", "generateTime": "2024-10-10T18:50:24.357027036Z" }, "purpose": "ENCRYPT_DECRYPT", "createTime": "2024-10-10T18:50:24.357027036Z", "versionTemplate": { "protectionLevel": "SOFTWARE", "algorithm": "GOOGLE_SYMMETRIC_ENCRYPTION" }, "destroyScheduledDuration": "2592000s" } Tcpdump output listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 18:13:51.220209 lo In IP 127.0.0.1.48873 > 127.0.0.53.53: 2086+ [1au] A? europe-west9-cloudkms.example.com. (62) 18:13:51.220230 lo In IP 127.0.0.1.48873 > 127.0.0.53.53: 24619+ [1au] AAAA? europe-west9-cloudkms.example.com. (62) 18:13:51.220669 ens4 Out IP 10.0.0.2.52121 > 169.254.169.254.53: 8885+ [1au] A? europe-west9-cloudkms.example.com. (62) 18:13:51.220784 ens4 Out IP 10.0.0.2.53041 > 169.254.169.254.53: 57748+ [1au] AAAA? europe-west9-cloudkms.example.com. (62) 18:13:51.229638 ens4 In IP 169.254.169.254.53 > 10.0.0.2.52121: 8885 1/0/1 A 10.0.0.100 (78) 18:13:51.229945 lo In IP 127.0.0.53.53 > 127.0.0.1.48873: 2086 1/0/1 A 10.0.0.100 (78) 18:13:51.230068 ens4 In IP 169.254.169.254.53 > 10.0.0.2.53041: 57748 0/1/1 (155) 18:13:51.230203 lo In IP 127.0.0.53.53 > 127.0.0.1.48873: 24619 0/1/1 (155) 18:13:51.230390 ens4 Out IP 10.0.0.2.59474 > 10.0.0.100.443: Flags [S], seq 1606150798, win 65320, options [mss 1420,sackOK,TS val 4135800856 ecr 0,nop,wscale 7], length 0 18:13:51.232565 ens4 In IP 10.0.0.100.443 > 10.0.0.2.59474: Flags [S.], seq 1041507402, ack 1606150799, win 65535, options [mss 1420,sackOK,TS val 2276748382 ecr 4135800856,nop,wscale 8], length 0 18:13:51.232583 ens4 Out IP 10.0.0.2.59474 > 10.0.0.100.443: Flags [.], ack 1, win 511, options [nop,nop,TS val 4135800859 ecr 2276748382], length 0 18:13:51.235494 ens4 Out IP 10.0.0.2.59474 > 10.0.0.100.443: Flags [P.], seq 1:518, ack 1, win 511, options [nop,nop,TS val 4135800862 ecr 2276748382], length 517 18:13:51.236571 ens4 In IP 10.0.0.100.443 > 10.0.0.2.59474: Flags [.], ack 518, win 267, options [nop,nop,TS val 2276748387 ecr 4135800862], length 0 18:13:51.239119 ens4 In IP 10.0.0.100.443 > 10.0.0.2.59474: Flags [P.], seq 1:1356, ack 518, win 267, options [nop,nop,TS val 2276748389 ecr 4135800862], length 1355 18:13:51.239140 ens4 Out IP 10.0.0.2.59474 > 10.0.0.100.443: Flags [.], ack 1356, win 501, options [nop,nop,TS val 4135800865 ecr 2276748389], length 0 18:13:51.240978 ens4 Out IP 10.0.0.2.59474 > 10.0.0.100.443: Flags [P.], seq 518:598, ack 1356, win 501, options [nop,nop,TS val 4135800867 ecr 2276748389], length 80 18:13:51.241266 ens4 Out IP 10.0.0.2.59474 > 10.0.0.100.443: Flags [P.], seq 598:684, ack 1356, win 501, options [nop,nop,TS val 4135800867 ecr 2276748389], length 86 18:13:51.241366 ens4 Out IP 10.0.0.2.59474 > 10.0.0.100.443: Flags [P.], seq 684:1646, ack 1356, win 501, options [nop,nop,TS val 4135800867 ecr 2276748389], length 962 18:13:51.242370 ens4 In IP 10.0.0.100.443 > 10.0.0.2.59474: Flags [.], ack 684, win 267, options [nop,nop,TS val 2276748392 ecr 4135800867], length 0 18:13:51.242619 ens4 In IP 10.0.0.100.443 > 10.0.0.2.59474: Flags [P.], seq 1356:1433, ack 1646, win 278, options [nop,nop,TS val 2276748393 ecr 4135800867], length 77 18:13:51.242730 ens4 Out IP 10.0.0.2.59474 > 10.0.0.100.443: Flags [P.], seq 1646:1677, ack 1433, win 501, options [nop,nop,TS val 4135800869 ecr 2276748393], length 31 18:13:51.248724 ens4 In IP 10.0.0.100.443 > 10.0.0.2.59474: Flags [.], ack 1677, win 278, options [nop,nop,TS val 2276748399 ecr 4135800869], length 0 18:13:51.296676 ens4 In IP 10.0.0.100.443 > 10.0.0.2.59474: Flags [P.], seq 1433:2357, ack 1677, win 278, options [nop,nop,TS val 2276748447 ecr 4135800869], length 924 18:13:51.297223 ens4 Out IP 10.0.0.2.59474 > 10.0.0.100.443: Flags [F.], seq 1677, ack 2357, win 501, options [nop,nop,TS val 4135800923 ecr 2276748447], length 0 18:13:51.298304 ens4 In IP 10.0.0.100.443 > 10.0.0.2.59474: Flags [P.], seq 2357:2381, ack 1678, win 278, options [nop,nop,TS val 2276748448 ecr 4135800923], length 24 18:13:51.298305 ens4 In IP 10.0.0.100.443 > 10.0.0.2.59474: Flags [F.], seq 2381, ack 1678, win 278, options [nop,nop,TS val 2276748448 ecr 4135800923], length 0 18:13:51.298336 ens4 Out IP 10.0.0.2.59474 > 10.0.0.100.443: Flags [R], seq 1606152476, win 0, length 0 18:13:51.298343 ens4 Out IP 10.0.0.2.59474 > 10.0.0.100.443: Flags [R], seq 1606152476, win 0, length 0
Check back on the tab 2 window and inspect the tcpdump information. You'll see that you were able to access the Cloud KMS regional endpoint through the PSC endpoint that you created and that the europe-west9 regional endpoint resolves privately to the Managed Cloud DNS zone you created. The relevant lines in the tcpdump output are highlighted above & referenced below:
18:13:51.229638 ens4 In IP 169.254.169.254.53 > 10.0.0.2.52121: 8885 1/0/1 A 10.0.0.100 (78)
(The GCP metadata server responds with the A record: 10.0.0.100- the load balancer IP. The DNS resolution is working correctly. europe-west9-cloudkms.example.com resolves to 10.0.0.100, which is your load balancer IP)
18:13:51.230390 ens4 Out IP 10.0.0.2.59474 > 10.0.0.100.443: Flags [S], seq 1606150798, win 65320, options [mss 1420,sackOK,TS val 4135800856 ecr 0,nop,wscale 7], length 0
(This shows the TCP handshake for the HTTPS connection to the load balancer IP)
In tab 3, client-vm
dig europe-west9-cloudkms.example.com
EXPECTED RESULT
; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> europe-west9-cloudkms.example.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7008 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;europe-west9-cloudkms.example.com. IN A ;; ANSWER SECTION: europe-west9-cloudkms.example.com. 300 IN A 10.0.0.100 ;; Query time: 12 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP) ;; WHEN: Fri Oct 11 20:03:00 UTC 2024 ;; MSG SIZE rcvd: 78
The output of the dig command shows that the custom URL we created for europe-west9-cloudkms.example.com correctly resolves to 10.0.0.100, which is your internal load balancer IP. Requests to europe-west9-cloudkms.example.com will be directed to your internal load balancer, which then forwards them to the KMS regional endpoint via Private Service Connect.
You can now close both SSH tabs to client-vm.
12. Cleanup steps
From a single Cloud Shell terminal delete lab components
# Set environment variables export PROJECT_ID=$(gcloud config get-value project) export PROJECT_NUMBER=$(gcloud projects describe ${PROJECT_ID} --format="value(projectNumber)") export REGION=europe-west9 export ZONE=europe-west9-a export VPC_NAME="consumer-vpc" export SUBNET_NAME="consumer-subnet-1" export NEG_NAME="l7psc-kms-neg" export BACKEND_SERVICE_NAME="l7-psc-kms" export CERT_NAME="my-ssl-cert" export PROXY_NAME="psc-http-proxy" export LB_RULE_NAME="kms-lb-rule" export DNS_ZONE_NAME="example-com-private-zone" # Delete DNS records and zone gcloud dns record-sets delete europe-west9-cloudkms.example.com. \ --zone=${DNS_ZONE_NAME} --type=A --quiet gcloud dns managed-zones delete ${DNS_ZONE_NAME} --quiet # Delete Load Balancer components gcloud compute forwarding-rules delete ${LB_RULE_NAME} --region=${REGION} --quiet gcloud compute target-https-proxies delete ${PROXY_NAME} --region=${REGION} --quiet gcloud compute url-maps delete l7-psc-url-map --region=${REGION} --quiet gcloud compute backend-services delete ${BACKEND_SERVICE_NAME} --region=${REGION} --quiet gcloud compute network-endpoint-groups delete ${NEG_NAME} --region=${REGION} --quiet # Delete SSL certificate gcloud compute ssl-certificates delete ${CERT_NAME} --region=${REGION} --quiet # Delete VM instance gcloud compute instances delete client-vm --zone=${ZONE} --quiet # Delete firewall rules gcloud compute firewall-rules delete allow-ssh-iap --quiet # Delete Cloud NAT and router gcloud compute routers nats delete europe-nat --router=crnat --region=${REGION} --quiet gcloud compute routers delete crnat --region=${REGION} --quiet # Delete subnets and VPC gcloud compute networks subnets delete ${SUBNET_NAME} --region=${REGION} --quiet gcloud compute networks subnets delete eu-west9-proxy-subnet --region=${REGION} --quiet gcloud compute networks delete ${VPC_NAME} --quiet # Schedule KMS key for deletion and provide instructions for keyring deletion gcloud kms keys remove-iam-policy-binding europe-key \ --location ${REGION} \ --keyring europe-kr \ --member serviceAccount:${PROJECT_NUMBER}-compute@developer.gserviceaccount.com \ --role roles/cloudkms.admin gcloud kms keys versions destroy 1 --location=${REGION} --keyring=europe-kr --key=europe-key # Disable services (optional, only if you want to completely disable these APIs) gcloud services disable compute.googleapis.com --force gcloud services disable servicedirectory.googleapis.com --force gcloud services disable dns.googleapis.com --force gcloud services disable cloudkms.googleapis.com --force # Clean up local files rm -f private.key server.csr server.crt csr_config.cnf startup.sh
13. Congratulations!
Congratulations on completing the codelab.