Access Gemini chat with python sdk via Private Service Connect endpoint

1. Overview

Vertex AI API can be accessed over the internet, however, in your enterprise you may want to access the Vertex AI API's privately without going over the internet. In this lab you will first access the Vertex Gemini chat API via python sdk running on a VM instance via public internet.

Then you will create a Private Service Connect endpoint to Google APIs, and change the traffic flow to use the private endpoint to connect to the Gemini chat API. The configurations will be a combination of Terraform, gcloud and console.

In this lab, you're going to be creating the following pattern.

Figure1.

2. Objective

In this lab you will learn how to perform the following task:

• Set up VM instance to use python sdk
• Connect to Gemini chat via python script
• Configure PSC endpoint to connect to Googleapis
• Verify connectivity path to Googleais
• Configure manual DNS entries

Self-paced environment setup

1. Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

• The Project name is the display name for this project's participants. It is a character string not used by Google APIs. You can always update it.
• The Project ID is unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference your Project ID (typically identified as PROJECT_ID). If you don't like the generated ID, you might generate another random one. Alternatively, you can try your own, and see if it's available. It can't be changed after this step and remains for the duration of the project.
• For your information, there is a third value, a Project Number, which some APIs use. Learn more about all three of these values in the documentation.

Caution: A project ID is globally unique and can't be used by anyone else after you've selected it. You are the only user of that ID. Even if a project is deleted, the ID can't be used again

Note: If you use a Gmail account, you can leave the default location set to No organization. If you use a Google Workspace account, choose a location that makes sense for your organization.

2. Next, you'll need to enable billing in the Cloud Console to use Cloud resources/APIs. Running through this codelab won't cost much, if anything at all. To shut down resources to avoid incurring billing beyond this tutorial, you can delete the resources you created or delete the project. New Google Cloud users are eligible for the $300 USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the Google Cloud Console, click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this codelab can be done within a browser. You do not need to install anything.

3. Task 1. Setup Environment with terraform

We will create a custom VPC with Firewall rules and subnet. Open the cloud console and select the project you will be using.

1. Open Cloud Shell located at the top of your console on the right, ensure you see the correct project id in Cloud Shell, confirm any prompts to allow access.
2. Create a folder called terraform-build and move to folder

mkdir terraform-build  && cd terraform-build

3. Create a main.tf and variable.tf file.

touch main.tf variable.tf 

4. Switch over to the Cloud Shell editor view. Select editor, ensure you allow any necessary prompts so the interface can load.
5. Once loaded navigate to, File > Open Folder and go to the /home/your-user-name/terraform-build and select Ok to open the folder in the editor.
6. Select the variable.tf file and add the following. Replace the your-project-id-here text with your actual project ID in quotes

variable "project_id" {
  type = string
  default = "your-project-id-here"
}

variable "network_id" {
  type = string
  default = "python-net"
}

7. Next open the main.tf file. We are going to add some terraform code to perform various actions as explained below.

Enable API's	resource "google_project_service" "default"
Create VPC called python-net	resource "google_compute_network" "default"
Add a subnet	resource "google_compute_subnetwork" "default"
Add two Firewall rules	resource "google_compute_firewall" "allow_icmp"resource "google_compute_firewall" "allow_ssh"

8. Copy and paste the following into the main .tf file.

resource "google_project_service" "default" {
  for_each = toset([
    "dns.googleapis.com",
    "aiplatform.googleapis.com",
    "servicedirectory.googleapis.com"
  ])

  service            = each.value
  disable_on_destroy = false
}

resource "google_compute_network" "default" {
  project                 = var.project_id
  name                    = var.network_id
  auto_create_subnetworks = false
  mtu                     = 1460
  routing_mode            = "GLOBAL"
}

resource "google_compute_subnetwork" "default" {
  name          = "vm1-subnet"
  ip_cidr_range = "10.0.11.0/24"
  region        = "us-east1"
  stack_type    = "IPV4_ONLY"
  network       = google_compute_network.default.id
}

resource "google_compute_firewall" "allow_icmp" {
  name    = "allow-icmp-${google_compute_network.default.name}"
  network = google_compute_network.default.id
  project = var.project_id

  allow {
    protocol = "icmp"
  }

  source_ranges = ["0.0.0.0/0"]
  target_tags   = ["allow-icmp"]
}

resource "google_compute_firewall" "allow_ssh" {
  name    = "allow-ssh-${google_compute_network.default.name}"
  network = google_compute_network.default.id
  project = var.project_id

  allow {
    protocol = "tcp"
    ports    = ["22"]
  }

  source_ranges = ["0.0.0.0/0"]
  target_tags   = ["allow-ssh"]
}

9. Switch back to Cloud Shell terminal make sure your are in the terraform-build directory cd terraform-build and run the following commands

terraform init

Initializes the working directory. This step downloads the providers required for the given configuration.

terraform plan

Generates an execution plan, showing what actions Terraform will take to deploy your infrastructure.

10. Now to create the resources run the terraform apply command and type yes to run.

Verify your setup: Check your environment for

1 VPC called python-net

1 subnet

2 firewall-rules

4. Task 2. Create NAT gateway and VMs with Terraform

We need to grant outbound external access to the internet so lets create a Cloud NAT gateway and attach it.

1. Open Cloud Shell, navigate to terraform-build folder and create the following files (total three files). We will edit these later on.

touch nat-vm.tf psc.tf dns.tf

2. Switch to Cloud Shell editor view and select the nat-vm.tf file and add the following Terraform code. This will create a NAT gateway and two VMs.

Terraform nat-vm.tf

resource "google_compute_router" "default" {
  name    = "py-outbound-nat"
  region  = "us-east1"
  network = google_compute_network.default.id

 bgp {
  asn = 64514
  }
}

resource "google_compute_router_nat" "default" {
  name                               = "py-outbound-nat-gw"
  router                             = google_compute_router.default.name
  region                             = google_compute_router.default.region
  nat_ip_allocate_option             = "AUTO_ONLY"
  source_subnetwork_ip_ranges_to_nat = "ALL_SUBNETWORKS_ALL_IP_RANGES"

  log_config {
    enable = true
    filter = "ERRORS_ONLY"
  }
}

resource "google_compute_instance" "vm1" {
  name         = "py-vm1"
  zone         = "us-east1-b"
  machine_type = "n2-standard-2"

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"
    }
  }

  network_interface {
    subnetwork = google_compute_subnetwork.default.id  
    stack_type = "IPV4_ONLY"
  }

  tags = ["allow-ssh", "allow-icmp"]

  metadata_startup_script = <<-EOF
    sudo apt-get update
    sudo apt-get install python3 python3-dev python3-venv -y
    sudo apt-get install tcpdump dnsutils -y
    sudo -i
    sudo mkdir -p ~/py-gem-env
    cd ~/py-gem-env
    python3 -m venv env
    source env/bin/activate
    pip install ipython google-cloud-aiplatform
  EOF
}

resource "google_compute_instance" "vm2" {
  name         = "py-vm2"
  zone         = "us-east1-b"
  machine_type = "n2-standard-2"

  boot_disk {
    initialize_params {
      image = "debian-cloud/debian-11"
    }
  }

  network_interface {
    subnetwork = google_compute_subnetwork.default.id   
    stack_type = "IPV4_ONLY"
  }

  tags = ["allow-ssh", "allow-icmp"]

  metadata_startup_script = <<-EOF
    sudo apt-get update
    sudo apt-get install python3 python3-dev python3-venv -y
    sudo apt-get install tcpdump dnsutils -y
    sudo -i
    sudo mkdir -p ~/py-gem-env
    cd ~/py-gem-env
    python3 -m venv env
    source env/bin/activate
    pip install ipython google-cloud-aiplatform
  EOF
}

3. Switch to Cloud Shell terminal, ensure your are in the terraform-build folder and run terraform plan this will show you that 4 items will be added, then run terraform apply and type yes to create the NAT gateway and two vm's.

Verify your setup: Check your environment for

1 NAT gateway

1 cloud router

VM named py-vm1

VM named py-vm2

5. Task 3. Configure VMs and test

1. Navigate to VM instances. Select the vm starting with py-vm1. Choose SSH.
2. Once you SSH into py-vm1, enable root by typing sudo -i
3. Activate your venv environment:

cd py-gem-env
source env/bin/activate

4. Now let's authenticate this to do some testing later on. Run the following command in the VM, press y when prompted.

gcloud auth application-default login

5. Next copy the url which appears in the starting with https:// open a new tab in your lab browser window and paste the url. Accept the prompts.
6. When you see the following select copy, switch back to the vm py-vm1 session and for Enter authorization code: paste the code you copied and press enter to authenticate.

7. Now let's do a quick test to see if we can connect to the Vertex Gemini API, this uses the us-central1-aiplatform.googleapis.com so we will do a dig to that address to see how the traffic routes.

dig us-central1-aiplatform.googleapis.com

8. You should see something similar (the address will differ). Note the path is via public IP addresses since the API is a public API. DON'T COPY

; <<>> DiG 9.16.48-Debian <<>> us-central1-aiplatform.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9117
;; flags: qr rd ra; QUERY: 1, ANSWER: 16, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;us-central1-aiplatform.googleapis.com. IN A

;; ANSWER SECTION:
us-central1-aiplatform.googleapis.com. 300 IN A 173.194.210.95
us-central1-aiplatform.googleapis.com. 300 IN A 173.194.211.95
us-central1-aiplatform.googleapis.com. 300 IN A 173.194.212.95
us-central1-aiplatform.googleapis.com. 300 IN A 173.194.213.95
us-central1-aiplatform.googleapis.com. 300 IN A 173.194.215.95
us-central1-aiplatform.googleapis.com. 300 IN A 173.194.216.95
us-central1-aiplatform.googleapis.com. 300 IN A 108.177.12.95
us-central1-aiplatform.googleapis.com. 300 IN A 108.177.13.95
us-central1-aiplatform.googleapis.com. 300 IN A 74.125.26.95

9. Now let's use python. Type ipython to activate the ipython interface.

ipython

10. Now copy and paste the following. This asks Gemini "What are all the colors of the Google logo" and "What is the color of the sky" . Replace enter-your-project-id-here with you project id in the quotes

import vertexai
from vertexai.generative_models import GenerativeModel, ChatSession

project_id = "enter-your-project-id-here" 
location = "us-central1" 

vertexai.init(project=project_id, location=location)
model = GenerativeModel("gemini-1.5-pro")
chat_session = model.start_chat()  

def get_chat_response(chat: ChatSession, prompt: str) -> str:
    text_response = []
    responses = chat.send_message(prompt, stream=True)
    for chunk in responses:
        text_response.append(chunk.text)
    return "".join(text_response)

prompt = "Hello."
print(get_chat_response(chat_session, prompt)) 

prompt = "What are all the colors of the Google logo?"
print(get_chat_response(chat_session, prompt)) 

prompt = "What color is the sky?"
print(get_chat_response(chat_session, prompt))

11. Press enter to run and see the result.
12. This request accessed Vertex via the public API.
13. Close off SSH session let's continue.

Note: You can modify the prompt question to something else.

Now configure py-vm2 with the same configs

14. Navigate to VM instances. Select the vm starting with py-vm2. Choose SSH.
15. Once you SSH into py-vm2-**** enable root by typing **sudo -i**
16. Activate your venv environment:

cd py-gem-env
source env/bin/activate

17. Now let's authenticate this to do some testing later on. Run the following command in the VM

gcloud auth application-default login

18. Next copy the url which appears in the starting with https:// open a new tab in your lab browser window and paste the url. Accept the prompts.
19. When you see the following select copy, switch back to the vm py-vm2 session and for Enter authorization code: paste the code you copied and press enter to authenticate.

20. Now let's do a quick test to see if we can connect to the Vertex Gemini API. This command will use 4 pings to the us-central1-aiplatform.googleapis.com so we will get a response from the public address of the API.

ping -c 4 us-central1-aiplatform.googleapis.com

21. We will come back to test this VM later on. Close off the SSH session and let's continue.

6. Task 4. Create the PSC endpoint to googleapis with Terraform

To enable private connectivity to our Vertex API endpoint we will create a Private Service Connect endpoint for Google APIs. This will allow us to use a private IP address we assign to route traffic to the Google APIs we need, in this case Vertex.

1. Open Cloud Shell in editor view if not open already. We are going to create the following:

• Create an IP for the PSC endpoint 192.168.255.250 (resource "google_compute_global_address" "default")
• Create a PSC endpoint to Google APIs (resource "google_compute_global_forwarding_rule" "default")

Open the psc.tf file in the terraform-build folder. Add the following code to file.

Terraform psc.tf

resource "google_compute_global_address" "default" {
  name         = "vertex-ip"
  purpose      = "PRIVATE_SERVICE_CONNECT"
  network      = google_compute_network.default.id
  address_type = "INTERNAL"
  address      = "192.168.255.250"
}

resource "google_compute_global_forwarding_rule" "default" {  
  name                  = "pscvertexgemini"
  target                = "all-apis"
  network               = google_compute_network.default.id
  ip_address            = google_compute_global_address.default.id
  load_balancing_scheme = ""
  }

2. Switch to Cloud Shell terminal, ensure your are in the terraform-build folder. Then run terraform init Then run terraform plan this will show you that 2 items will be added,
   then run terraform apply and type yes to create the IP and PSC Google APIs endpoint.
3. Verify endpoint exist

gcloud compute addresses list --filter="name=( 'vertex-ip' ...)"

gcloud compute forwarding-rules describe pscvertexgemini --global

7. Task 5. Verify endpoint connectivity via IP address

Let's connect using the private endpoint to connect to Gemini.

1. Go to VM Instance py-vm1. Select SSH and SSH into the VM
2. Gain root access by typing sudo -i
3. We will just use this individual instance to test the PSC endpoint so we will modify the host file with the following entry

echo 192.168.255.250 us-central1-aiplatform.googleapis.com >> /etc/hosts

cat /etc/hosts

Note: This will add a static entry which will map the us-central1-aiplatform.googleapis.com address to the PSC endpoint IP.

4. Check the connectivity path to us-central1-aiplatform.googleapis.com using the ping command. This will ping the IP address you entered in the host files. This is a PSC endpoint and your pings will be unsuccessful.

ping -c 2 us-central1-aiplatform.googleapis.com

5. Go back to the console and open another instance of VM Instance py-vm1. Select SSH and SSH into the VM
6. Gain root access by typing sudo -i
7. Run the follow command to see the connectivity in a TCP dump

sudo tcpdump -i any port 53 -n or host us-central1-aiplatform.googleapis.com

8. Now switch back to the first SSH instance of VM Instance py-vm1
9. Activate the env using

cd py-gem-env
source env/bin/activate

10. Now let's test python. Type ipython to activate the ipython interface. This time traffic will flow through the PSC endpoint.

ipython

11. Now copy and paste the following. This ask Gemini "What are all the colors of the Google logo" and "Describe Niagara Falls". Replace enter-your-project-id-here with you project id in the quotes

import vertexai
from vertexai.generative_models import GenerativeModel, ChatSession

project_id = "enter-your-project-id-here" 
location = "us-central1" 

vertexai.init(project=project_id, location=location)
model = GenerativeModel("gemini-1.5-pro")
chat_session = model.start_chat()  # Corrected line

def get_chat_response(chat: ChatSession, prompt: str) -> str:
    text_response = []
    responses = chat.send_message(prompt, stream=True)
    for chunk in responses:
        text_response.append(chunk.text)
    return "".join(text_response)

prompt = "Hello."
print(get_chat_response(chat_session, prompt)) 

prompt = "What are all the colors of the Google logo?"
print(get_chat_response(chat_session, prompt)) 

prompt = "Describe Niagara Falls"
print(get_chat_response(chat_session, prompt))

12. Press enter to run and see the result.
13. Switch back over to the second instance of VM Instance py-vm1. You should see the result of the TCPDUMP. You will notice In and Out with the IP addresses of the VM and also the PSC endpoint IP address to connect to us-central1-aiplatform.googleapis.com

22:21:55.032433 ens4  Out IP 10.0.11.18.57114 > 192.168.255.250.443: Flags [.], ack 8606, win 501, options [nop,nop,TS val 1797790182 ecr 2593039209], length 0
22:21:55.468285 ens4  In  IP 192.168.255.250.443 > 10.0.11.18.57114: Flags [P.], seq 8606:8991, ack 5785, win 296, options [nop,nop,TS val 2593039645 ecr 1797790182], length 385
22:21:55.468320 ens4  Out IP 10.0.11.18.57114 > 192.168.255.250.443: Flags [.], ack 8991, win 501, options [nop,nop,TS val 1797790618 ecr 2593039645], length 0

14. Close all SSH sessions to VM Instance py-vm1

8. Task 6. Create Manual DNS entry to googleapis with Terraform (Optional)

You can create a manual DNS entry to point to the PSC endpoint using private DNS. This would affect all the networks you assign to it.

1. Navigate to Network services and select Cloud DNS.
2. In zones you should see an automatically created zone for the Private Service Connect for Google APIs, with zone type service directory. This can be used to connect to the PSC endpoint with the format **SERVICE-ENDPOINT.p.googleapis.com Example: aiplatform-pscvertexgemini.p.googleapis.com
3. In this case we want to manually create a private DNS entry. The config will be as follows

• Create a private DNS zone named "googleapis-private" for "googleapis.com" and restrict it to the "python-net" network.
• Add an A record to map "googleapis.com" to the IP address "192.168.255.250".
• Add a CNAME record to redirect all subdomains of "googleapis.com" (e.g., www.googleapis.com) to "googleapis.com".

4. Open Cloud Shell in editor view if not open already. Open the dns.tf file in the terraform-build folder. Add the following code to file.

Terraform dns.tf

resource "google_dns_managed_zone" "private_zone" {
  name        = "googleapis-private"
  dns_name    = "googleapis.com."  
  visibility  = "private"
  project     = var.project_id     

  private_visibility_config {
    networks {
      network_url = google_compute_network.default.id  
    }
  }
}

resource "google_dns_record_set" "a_record" {
  name    = "googleapis.com."  
  type    = "A"
  ttl     = 300
  managed_zone = google_dns_managed_zone.private_zone.name
  project = var.project_id    

  rrdatas = ["192.168.255.250"]
}

resource "google_dns_record_set" "cname_record" {
 name    = "*.googleapis.com."
 type    = "CNAME"
 ttl     = 300
 managed_zone = google_dns_managed_zone.private_zone.name
 project = var.project_id    

 rrdatas = ["googleapis.com."]  
}

5. Switch to Cloud Shell terminal, ensure your are in the terraform-build folder. Then run terraform plan this will show you what items will be added,
   Then run terraform apply and type yes to create the private DNS entry.
6. You should see a set up with an A record and CNAME like this see
7. Next we verify connectivity with these changes on py-vm2

9. Task 7. Verify endpoint connectivity via IP address (Optional)

Let's connect using the private endpoint to connect to Gemini.

Note: No manual configurations are necessary on the VM.

1. Go to VM Instance py-vm2. Select SSH and SSH into the VM
2. Gain root access by typing sudo -i
3. Check the connectivity path to us-central1-aiplatform.googleapis.com using the ping command. This will ping the IP address in the private DNS, A record for googleapis. This IP a PSC endpoint and your pings will be unsuccessful.

ping -c 2 us-central1-aiplatform.googleapis.com

4. Check the connectivity path with a ping using the automatically created DNS entry for PSC Google APIs with aiplatform-pscvertexgemini.p.googleapis.com. This points to the IP address of the PSC endpoint and your pings will be unsuccessful.

ping -c 2 aiplatform-pscvertexgemini.p.googleapis.com

5. Check the connectivity path to us-central1-aiplatform.googleapis.com using the dig command. This should be the IP address of the PSC endpoint.

dig us-central1-aiplatform.googleapis.com

6. Go back to the console and open another instance of VM Instance py-vm2. Select SSH and SSH into the VM
7. Gain root access by typing sudo -i
8. Run the follow command to see the connectivity in a TCP dump

sudo tcpdump -i any port 53 -n or host us-central1-aiplatform.googleapis.com

9. Now switch back to the first SSH instance of VM Instance py-vm2
10. Activate the env using

cd py-gem-env
source env/bin/activate

11. Now let's test python. Type ipython to activate the ipython interface.

ipython

12. Now copy and paste the following. This ask Gemini "What are all the colors of the Google logo" and "What are two features of Gemini pro". Replace enter-your-project-id-here with you project id in the quotes

import vertexai
from vertexai.generative_models import GenerativeModel, ChatSession

project_id = "enter-your-project-id-here" 
location = "us-central1" 

vertexai.init(project=project_id, location=location)
model = GenerativeModel("gemini-1.5-pro")
chat_session = model.start_chat()  # Corrected line

def get_chat_response(chat: ChatSession, prompt: str) -> str:
    text_response = []
    responses = chat.send_message(prompt, stream=True)
    for chunk in responses:
        text_response.append(chunk.text)
    return "".join(text_response)

prompt = "Hello."
print(get_chat_response(chat_session, prompt)) 

prompt = "What are all the colors of the Google logo?"
print(get_chat_response(chat_session, prompt)) 

prompt = "What are two features of Gemini pro"
print(get_chat_response(chat_session, prompt))

13. Press enter to run and see the result.
14. Switch back over to the second instance of VM Instance py-vm2. You should see the result of the TCPDUMP. You will notice in and out and the IP address of the VM is using the PSC endpoint IP address to connect to us-central1-aiplatform.googleapis.com

Close all SSH sessions to VM Instance py-vm2

10. Clean Up

1. Go to Cloud Shell make sure you are in the terraform-build directory cd terraform-build and run the following command terraform destroy and type yes all the resources you created in your project with Terraform will be removed.

11. Congratulations

Congratulations, you've successfully connected to Vertex Gemini chat using both public API address and privately using Private Service Connect Endpoint for Google APIs. This functionality can extend private API connectivity into your on-prem/other cloud environment which are connected via (Interconnect, Cross-Cloud Interconnect and VPC).

Next steps / Learn more

You can read more about Vertex AI networking

Codelab: Access Anthropic Claude on Vertex AI with python sdk via Private Service Connect endpoint

Take your next lab

Continue your quest with Google Cloud, and check out these other Google Cloud Skills Boost labs:

• Networking Fundamentals in Google Cloud
• Generative AI Explorer - Vertex AI