Access Anthropic Claude on Vertex AI with python sdk via Private Service Connect endpoint

1. Overview

Vertex AI API can be accessed over the internet, however, in your enterprise you may want to access the Vertex AI API's privately without going over the internet. In this lab you will first access the Anthropic Claude on Vertex via python sdk running on a VM instance via public internet.

Then you will create a Private Service Connect endpoint to Googleapis, and change the traffic flow to use the private endpoint to connect to the Vertex API.

In this lab, you're going to be creating the following pattern.

Figure1.

2. Objectives

In this lab you will learn how to perform the following task:

• Set up VM instance to use python sdk
• Connect to Anthropic Claude chat via python script
• Configure PSC endpoint to connect to Googleapis
• Configure manual DNS entries
• Verify connectivity path to Googleais

Lab setup

Self-paced environment setup

1. Sign-in to the Google Cloud Console and create a new project or reuse an existing one. If you don't already have a Gmail or Google Workspace account, you must create one.

• The Project name is the display name for this project's participants. It is a character string not used by Google APIs. You can always update it.
• The Project ID is unique across all Google Cloud projects and is immutable (cannot be changed after it has been set). The Cloud Console auto-generates a unique string; usually you don't care what it is. In most codelabs, you'll need to reference your Project ID (typically identified as PROJECT_ID). If you don't like the generated ID, you might generate another random one. Alternatively, you can try your own, and see if it's available. It can't be changed after this step and remains for the duration of the project.
• For your information, there is a third value, a Project Number, which some APIs use. Learn more about all three of these values in the documentation.

Caution: A project ID is globally unique and can't be used by anyone else after you've selected it. You are the only user of that ID. Even if a project is deleted, the ID can't be used again

Note: If you use a Gmail account, you can leave the default location set to No organization. If you use a Google Workspace account, choose a location that makes sense for your organization.

2. Next, you'll need to enable billing in the Cloud Console to use Cloud resources/APIs. Running through this codelab won't cost much, if anything at all. To shut down resources to avoid incurring billing beyond this tutorial, you can delete the resources you created or delete the project. New Google Cloud users are eligible for the $300 USD Free Trial program.

Start Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you will be using Google Cloud Shell, a command line environment running in the Cloud.

From the Google Cloud Console, click the Cloud Shell icon on the top right toolbar:

It should only take a few moments to provision and connect to the environment. When it is finished, you should see something like this:

This virtual machine is loaded with all the development tools you'll need. It offers a persistent 5GB home directory, and runs on Google Cloud, greatly enhancing network performance and authentication. All of your work in this codelab can be done within a browser. You do not need to install anything.

3. Task 1. Setup Environment

Please be aware that some resources you enable are going to incur some cost if you are not using the promotional tier. To reduce cost ensure the resources are destroyed upon completion of the lab. We recommend checking your billing and making sure the exercise is acceptable for you.

We will create a custom VPC with Firewall rules. If you have a VPC and Project already you can skip this part.

Open Cloud Shell located at the top of your console on the right. And configure as follows:

1. Enable some of the apis we will use in this lab

gcloud services enable dns.googleapis.com
gcloud services enable aiplatform.googleapis.com
gcloud services enable servicedirectory.googleapis.com

2. Set some variables. These variables are the Project ID and Network ID of the custom VPC (you will create the VPC in step 4).

projectid=$(gcloud config get-value project)
networkid=anthropic-net 
echo $projectid
echo $networkid
clear

3. Now create a custom VPC called anthropic-net.

gcloud compute networks create $networkid \
--project=$projectid \
--subnet-mode=custom \
--mtu=1460 \
--bgp-routing-mode=global

4. Create the vm1-subnet in the new VPC

gcloud compute networks subnets create vm-subnet \
--project=$projectid --range=10.0.88.0/24 \
--stack-type=IPV4_ONLY --network=$networkid \
--region=us-east1

5. Add ICMP firewall rule to your VPC

gcloud compute firewall-rules create $networkid-allow-icmp --project=$projectid \
--network=$networkid \
--description="Allows ICMP connections from any source to any instance on the network." \
--direction=INGRESS \
--priority=65534 \
--source-ranges=0.0.0.0/0 \
--action=ALLOW \
--rules=icmp

6. Add SSH firewall rule to your VPC

gcloud compute firewall-rules create $networkid-allow-ssh \
--project=$projectid \
--network=$networkid \
--description="Allows TCP connections from any source to any instance on the network using port 22." \
--direction=INGRESS --priority=65534 \
--source-ranges=0.0.0.0/0 --action=ALLOW \
--rules=tcp:22

4. Task 2. Enable Anthropic in Vertex model garden

We need to grant outbound external access to the internet so lets create a Cloud NAT gateway and attach it.

1. Go to Vertex AI and select Model Garden
2. Search for Anthropic and select Claude 3.5 Sonnet

3. Select Enable, you will be required to fill out some info. Fill out form an select Next
4. On the final page select Agree to enable Claude 3.5 Sonnet

Note: The process should be the same for any model

5. Task 3. Create NAT gateway and VMs

We need to grant outbound external access to the internet so lets create a Cloud NAT gateway and attach it.

In Cloud Shell use the following commands

1. Create the Cloud NAT.

gcloud compute routers create anthro-out-nat \
    --network $networkid \
    --region us-east1 

2. Create the Cloud NAT gateway.

gcloud compute routers nats create anthro-out-nat-gw \
    --router-region us-east1 \
    --router anthro-out-nat \
    --nat-all-subnet-ip-ranges \
    --auto-allocate-nat-external-ips

Now let's create one VMs to access the Anthropic on Vertex AI via Python SDK.

We are going to create a vm to test with the following packages installed

1. In the same cloud shell session create anthro-vm using the following.

gcloud compute instances create anthro-vm \
--project=$projectid \
--zone=us-east1-b \
--network-interface=stack-type=IPV4_ONLY,subnet=vm-subnet,no-address,network=$networkid \
--metadata startup-script="#! /bin/bash    
      sudo apt-get update
      sudo apt-get install python3 python3-dev python3-venv -y
      sudo apt-get install tcpdump dnsutils -y
      sudo -i
      sudo mkdir -p ~/py-anthro-env
      cd ~/py-anthro-env
      python3 -m venv env
      source env/bin/activate
      pip install -U ipython google-cloud-aiplatform 'anthropic[vertex]'"

You should see a VM created with no public IP addresses. Now let's configure the VM's

6. Task 4. Configure VM and test

1. Navigate to VM instances. Select the vm starting with anthro-vm. Choose SSH.
2. Once you SSH into anthro-vm, enable root by typing sudo -i
3. Activate your venv environment:

cd py-anthro-env
source env/bin/activate

4. Now let's authenticate this to do some testing later on. Run the following command in the VM, press y when prompted.

gcloud auth application-default login

5. Next copy the url which appears in the starting with https:// open a new tab in your lab browser window and paste the url. Accept the prompts.
6. When you see the following select copy, switch back to the vm anthro-vm session and for Enter authorization code: paste the code you copied and press enter to authenticate.

7. Now let's do a quick test to see if we can connect to the Vertex Gemini API, this uses the us-east5-aiplatform.googleapis.com so we will do a dig to that address to see how the traffic routes.

dig us-east5-aiplatform.googleapis.com

8. You should see something similar (the address will differ). Note the path is via public IP addresses since the API is a public API.

; <<>> DiG 9.16.48-Debian <<>> us-east5-aiplatform.googleapis.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9117
;; flags: qr rd ra; QUERY: 1, ANSWER: 16, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;us-east5-aiplatform.googleapis.com. IN A

;; ANSWER SECTION:
us-east5-aiplatform.googleapis.com. 300 IN A 173.194.210.95
us-east5-aiplatform.googleapis.com. 300 IN A 173.194.211.95
us-east5-aiplatform.googleapis.com. 300 IN A 173.194.212.95
us-east5-aiplatform.googleapis.com. 300 IN A 173.194.213.95
us-east5-aiplatform.googleapis.com. 300 IN A 173.194.215.95
us-east5-aiplatform.googleapis.com. 300 IN A 173.194.216.95
us-east5-aiplatform.googleapis.com. 300 IN A 108.177.12.95
us-east5-aiplatform.googleapis.com. 300 IN A 108.177.13.95
us-east5-aiplatform.googleapis.com. 300 IN A 74.125.26.95

9. Now let's use python. Type ipython to activate the ipython interface.

ipython

10. Now copy and paste the following. This asks Claude "Where is the Olympics being held in 2024!" . p.s. Replace YOUR-Project-ID-Here with the name of your project id

from anthropic import AnthropicVertex

project_id = "YOUR-Project-ID-Here"
region = "us-east5"

client = AnthropicVertex(project_id=project_id, region=region)

message = client.messages.create(
    model="claude-3-5-sonnet@20240620",
    max_tokens=100,
    messages=[
        {
            "role": "user",
            "content": "Where is the olympics being held in 2024!",
        }
    ],
)
print(message)

11. Press enter to run and see the result.
12. This request accessed Anthropic via the Vertex public API.
13. Close off SSH session let's continue.

Note: You can modify the prompt question to something else.

7. Task 5. Create the PSC endpoint to googleapis

To enable private connectivity to our Vertex API endpoint we will create a Private Service Connect endpoint for googleapis. This will allow us to use a private IP address we assign to route traffic to the googleapis we need, in this case Vertex Gemini.

1. Open Cloud Shell if not open already. Create an IP for the PSC endpoint. We will use 192.168.255.230 in this case.

gcloud compute addresses create anthro-ip \
    --global \
    --purpose=PRIVATE_SERVICE_CONNECT \
    --addresses=192.168.255.230 \
    --network=$networkid

2. Verify IP created

gcloud compute addresses list --filter="name=( 'anthro-ip' ...)"

3. Next create the PSC endpoint

gcloud compute forwarding-rules create pscanthrovertex \
    --global \
    --network=$networkid \
    --address=anthro-ip \
    --target-google-apis-bundle=all-apis

4. This will create an endpoint and a service directory entry. Verify endpoint exist

gcloud compute forwarding-rules describe pscanthrovertex --global

8. Task 6. Verify endpoint connectivity via automatic service directory zone

Let's connect using the private endpoint to connect to Gemini.

1. Go to VM Instance anthro-vm1. Select SSH and SSH into the VM
2. Gain root access by typing sudo -i
3. Check the connectivity path to aiplatform-pscanthrovertex.p.googleapis.com using the dig command. You should see the IP of the PSC endpoint

dig aiplatform-pscanthrovertex.p.googleapis.com

9. Task 7. Create Manual DNS entry to googleapis (Optional)

You can create a manual DNS entry to point to the PSC endpoint using private DNS. This would affect all the networks you assign to it.

1. Navigate to Network services and select Cloud DNS.
2. In zones you should see an automatically created zone for the Private Service Connect for Google APIs, with zone type service directory. This can be used to connect to the PSC endpoint with the format **SERVICE-ENDPOINT.p.googleapis.com Example aiplatform-pscvertexgemini.p.googleapis.com
3. In this case we want to manually create a private DNS entry. Go to Cloud DNS and select Create Zone
4. Configure as follows

Configuration	Name
Zone type	Private
Zone name	googleapis-private
DNS name	googleapis.com
Add Networks (select done when added)	anthropic-net
To complete select	Create

5. In the Zone details area select Add standard to add an A record

Configuration	Name
Resource record type	A
IPv4 address (add IP address of endpoint)	192.168.255.230
To complete select	Create

6. In the Zone detail area select Add standard to add a CNAME record

Configuration	Name
DNS name	*
Record resource type	CNAME
Canonical name 1	googleapis.com
To complete select	Create

7. You should see a set up with an A record and CNAME like this see
8. Next we verify connectivity with these changes on anthro-vm

10. Task 8. Verify endpoint connectivity via IP address (Optional)

Let's connect using the private endpoint to connect to Gemini.

Note: No manual configurations are necessary on the VM.

1. Go to VM Instance anthro-vm. Select SSH and SSH into the VM
2. Gain root access by typing sudo -i
3. Check the connectivity path to us-east5-aiplatform.googleapis.com using the ping command. This will ping the IP address in the private DNS, A record for googleapis. This IP a PSC endpoint and your pings will be unsuccessful.

ping -c 2 us-east5-aiplatform.googleapis.com

4. Check the connectivity path with a ping using the automatically created DNS entry for PSC Google APIs with aiplatform-pscanthrovertex.p.googleapis.com. This points to the IP address of the PSC endpoint and your pings will be unsuccessful.

ping -c 2 aiplatform-pscanthrovertex.p.googleapis.com

5. Check the connectivity path to us-east5-aiplatform.googleapis.com using the dig command. This should be the IP address of the PSC endpoint.

dig us-east5-aiplatform.googleapis.com

6. Go back to the console and open another instance of VM Instance anthro-vm. Select SSH and SSH into the VM
7. Gain root access by typing sudo -i
8. Run the follow command to see the connectivity in a TCP dump

sudo tcpdump -i any port 53 -n or host us-east5-aiplatform.googleapis.com

9. Now switch back to the first SSH instance of VM Instance anthro-vm
10. Activate the env using

cd py-gem-env
source env/bin/activate

11. Now let's test python. Type ipython to activate the ipython interface.

ipython

12. Now copy and paste the following. This asks Claude "What is a roti?". p.s. Replace YOUR-Project-ID-Here with the name of your project id

from anthropic import AnthropicVertex

project_id = "YOUR-Project-ID-Here"
region = "us-east5"

client = AnthropicVertex(project_id=project_id, region=region)

message = client.messages.create(
    model="claude-3-5-sonnet@20240620",
    max_tokens=100,
    messages=[
        {
            "role": "user",
            "content": "What is a roti?",
        }
    ],
)
print(message)

13. Press enter to run and see the result.
14. Switch back over to the second instance of VM Instance anthro-vm. You should see the result of the TCPDUMP. You will notice in and out and the IP address of the VM is using the PSC endpoint IP address to connect to us-east5-aiplatform.googleapis.com

Close all SSH sessions to VM Instance anthro-vm

11. Congratulations

Congratulations, you've successfully connected to Anthropic on Vertex using both public API address and privately using Private Service Connect Endpoint for Googleapis. This functionality can extend private API connectivity into your on-prem/other cloud environment which are connected via (Interconnect, Cross-Cloud Interconnect and VPC).

Clean up

gcloud compute instances delete anthro-vm --zone=us-east1-b --quiet

gcloud compute routers nats delete anthro-out-nat-gw --router=anthro-out-nat --region=us-east1 --quiet

gcloud compute routers delete anthro-out-nat --region=us-east1 --quiet

gcloud compute firewall-rules delete anthropic-net-allow-icmp anthropic-net-allow-ssh --quiet

gcloud compute forwarding-rules delete pscanthrovertex --global

gcloud dns record-sets delete googleapis.com. --zone googleapis-private --type A

gcloud dns record-sets delete *.googleapis.com --zone googleapis-private --type CNAME

gcloud dns managed-zones delete googleapis-private

gcloud compute addresses delete anthro-ip --global


gcloud compute networks subnets delete vm-subnet --region=us-east1 --quiet

gcloud compute networks delete anthropic-net --quiet

You can read more about Vertex AI networking