Deploy on Amazon EKS
Tenet PII Engine 0.5.4 runs in your own EKS cluster, with no internet route. Your text and keys stay in your VPC. Detection and rehydration run as separate Deployments, each with its own IAM role. Callers reach them through Amazon VPC Lattice and authenticate with their own IAM role, so there are no shared tokens, and every rehydration and erasure is audited with the caller’s identity.
Use Graviton4 nodes for now
Use c8g.xlarge, not c9g.xlarge. On Graviton5, the VPC CNI network policy agent can drop an update to a network policy, and traffic it should allow then stays blocked until the next policy change. In our testing, 7 of 18 new callers were blocked on c9g and none of 12 on c8g. AWS tracks this as aws-network-policy-agent issue 686. It fails closed, so nothing is exposed, and it doesn’t affect ECS. We tested c8g with 0.4.2, and no release since changes the cluster network.
Before you start
- Subscribe to the product in AWS Marketplace. The detect Deployment refuses to start if this account isn’t subscribed.
- Create the node group. EC2 managed nodes on arm64, not Fargate, on Kubernetes 1.29 or later (tested on 1.37). With rehydration on, use four c8g.xlarge nodes, two in each of two Availability Zones: each detect pod fills a node, and rehydrate needs a free node in each zone. That’s 16 vCPUs, so check your On-Demand vCPU quota first, and leave headroom if your node group updates add a surge node. VPC Lattice isn’t offered in every zone, for example use1-az3 in us-east-1.
- Turn on network policy. Set enableNetworkPolicy to true on the VPC CNI add-on, or use Calico. Without an enforcer, the chart’s policies do nothing.
- Add an IAM OIDC provider for the cluster. AWS only accepts metering through IAM roles for service accounts, not EKS Pod Identity or the node role.
- Add VPC endpoints. Interface endpoints with private DNS for kms, sts, metering-marketplace, ecr.api, ecr.dkr, logs, vpc-lattice and tagging, and an S3 gateway endpoint. With no internet route, the nodes also need ec2 and eks, plus eks-auth if you use EKS Pod Identity add-ons. Give the interface endpoints a security group that allows 443 from your VPC. Detect doesn’t start if it can’t reach metering-marketplace. Outside us-east-1, also add three interface endpoints with private DNS whose service Region is us-east-1: ecr.api, ecr.dkr and s3. AWS Marketplace serves the image and chart only from us-east-1, and these reach it there with no NAT. Admin and PowerUser roles can create them; a role limited to specific actions also needs vpce:AllowMultiRegion.
- Create a customer-managed KMS key.
- Create the vault database. Aurora PostgreSQL (we test with Serverless v2, 16.15), encrypted with the KMS key, with a database named tenet: the URL in values.yaml names it. Allow the cluster security group to reach it on 5432. An RDS-managed master password rotates every 7 days, which leaves the Kubernetes secret below stale, so sync it with a tool such as External Secrets Operator or use a database user with a password you manage.
- Create the audit bucket. S3 with Object Lock and default encryption with the same KMS key. Use COMPLIANCE mode in production. For a trial, GOVERNANCE lets you delete it afterwards. Don’t reuse the audit bucket from the ECS templates: its policy admits only the ECS roles, so every rehydrate and erase call would return 503.
Set up VPC Lattice
Once per cluster:
- Install the Gateway API CRDs (v1.5.0 or later, standard channel) and the AWS Gateway API Controller v2.1.4, with its own IAM role for service accounts and the VPC Lattice permissions its documentation lists. In a cluster with no internet route, copy its image into your own ECR first. The controller needs the vpc-lattice and tagging endpoints: without the tagging endpoint, no VPC Lattice service appears and the pods restart while they wait for it.
- Create the Gateway. It becomes a VPC Lattice service network, and the controller associates the cluster’s VPC with it. A VPC can join only one service network, so if yours already has one, name the Gateway after it. Wait for the Gateway to show Programmed.
apiVersion: gateway.networking.k8s.io/v1kind: GatewayClassmetadata:name: amazon-vpc-latticespec:controllerName: application-networking.k8s.aws/gateway-api-controller---apiVersion: gateway.networking.k8s.io/v1kind: Gatewaymetadata:name: tenet-piinamespace: lattice-systemspec:gatewayClassName: amazon-vpc-latticelisteners:- name: httpsprotocol: HTTPSport: 443allowedRoutes:namespaces: { from: All }
- Let VPC Lattice reach the pods on 19991, in the security group the pods use: the node security group, or the pod security group if you use security groups for pods. The chart’s network policies then admit only lattice.peerCidrs, which defaults to 169.254.171.0/24 and 129.224.0.0/17. If the last command prints other ranges, set lattice.peerCidrs to them.
LATTICE_PL=$(aws ec2 describe-managed-prefix-lists \--filters Name=prefix-list-name,Values=com.amazonaws.$AWS_REGION.vpc-lattice \--query 'PrefixLists[0].PrefixListId' --output text)aws ec2 authorize-security-group-ingress --group-id <node-security-group-id> \--ip-permissions "IpProtocol=tcp,FromPort=19991,ToPort=19991,PrefixListIds=[{PrefixListId=$LATTICE_PL}]"aws ec2 get-managed-prefix-list-entries --prefix-list-id "$LATTICE_PL" --query 'Entries[].Cidr' --output text
Create the IAM roles
Create two roles, each trusting your cluster’s OIDC provider for one service account: system:serviceaccount:tenet-pii:tenet-pii-enterprise-detect and system:serviceaccount:tenet-pii:tenet-pii-enterprise-rehydrate. The chart uses these names whatever you call the release, and AWS Marketplace meters through the detect one. If your key policy doesn’t delegate to IAM, add the same KMS permissions there.
{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Principal": { "Federated": "arn:aws:iam::<account>:oidc-provider/<oidc-issuer-host-and-path>" },"Action": "sts:AssumeRoleWithWebIdentity","Condition": { "StringEquals": {"<oidc-issuer-host-and-path>:aud": "sts.amazonaws.com","<oidc-issuer-host-and-path>:sub": "system:serviceaccount:tenet-pii:tenet-pii-enterprise-detect"} }}]}
The rehydrate role’s trust policy is the same, with the rehydrate service account as the subject.
Both roles look up their own VPC Lattice service at startup. Only detect reports usage to AWS Marketplace, so only its role allows aws-marketplace:MeterUsage. Detect also writes an audit record for each erasure, under erasure/ in the audit bucket.
Detect creates the vault’s encryption keys but can never decrypt a stored value: only rehydrate can unwrap the private key. Keep detect’s two Deny statements, and add them to the key policy as well, so no permission granted to the role later can undo them.
{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Action": "aws-marketplace:MeterUsage", "Resource": "*" },{ "Effect": "Allow", "Action": "vpc-lattice:ListServices", "Resource": "*" },{"Effect": "Allow","Action": "kms:Encrypt","Resource": "<kms-key-arn>","Condition": { "StringEquals": {"kms:EncryptionContext:tenet:purpose": ["pii-vault-lookup-key", "pii-vault-private-key"],"kms:EncryptionContext:tenet:deployment": "<deployment-id>"} }},{"Effect": "Allow","Action": "kms:Decrypt","Resource": "<kms-key-arn>","Condition": { "StringEquals": {"kms:EncryptionContext:tenet:purpose": "pii-vault-lookup-key","kms:EncryptionContext:tenet:deployment": "<deployment-id>"} }},{"Effect": "Deny","Action": "kms:Decrypt","Resource": "*","Condition": { "StringEquals": { "kms:EncryptionContext:tenet:purpose": "pii-vault-private-key" } }},{ "Effect": "Deny", "Action": ["kms:ReEncrypt*", "kms:CreateGrant"], "Resource": "*" },{"Effect": "Allow","Action": "kms:GenerateDataKey","Resource": "<kms-key-arn>","Condition": { "StringEquals": { "kms:ViaService": "s3.<region>.amazonaws.com" } }},{ "Effect": "Allow", "Action": "s3:GetBucketObjectLockConfiguration", "Resource": "arn:aws:s3:::<audit-bucket>" },{ "Effect": "Allow", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::<audit-bucket>/erasure/*" }]}
{"Version": "2012-10-17","Statement": [{ "Effect": "Allow", "Action": "vpc-lattice:ListServices", "Resource": "*" },{"Effect": "Allow","Action": "kms:Decrypt","Resource": "<kms-key-arn>","Condition": { "StringEquals": {"kms:EncryptionContext:tenet:purpose": ["pii-vault-lookup-key", "pii-vault-private-key"],"kms:EncryptionContext:tenet:deployment": "<deployment-id>"} }},{"Effect": "Allow","Action": "kms:GenerateDataKey","Resource": "<kms-key-arn>","Condition": { "StringEquals": { "kms:ViaService": "s3.<region>.amazonaws.com" } }},{ "Effect": "Allow", "Action": "s3:GetBucketObjectLockConfiguration", "Resource": "arn:aws:s3:::<audit-bucket>" },{ "Effect": "Allow", "Action": "s3:PutObject", "Resource": "arn:aws:s3:::<audit-bucket>/rehydrate/*" }]}
Install
Pull the chart with the account that subscribed:
aws ecr get-login-password --region us-east-1 | \helm registry login 709825985650.dkr.ecr.us-east-1.amazonaws.com --username AWS --password-stdinhelm pull oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/tenet-labs/tenet-pii-enterprise --version 0.5.4
Create the vault credentials:
kubectl create namespace tenet-pii# DB_USER and DB_PASSWORD: a database user that can create tables in the tenet databaseprintf %s "$DB_USER" > user.txt; printf %s "$DB_PASSWORD" > password.txtkubectl -n tenet-pii create secret generic tenet-vault-db \--from-file=username=user.txt --from-file=password=password.txtrm user.txt password.txt
The network policies need your Region’s S3 address ranges, for audit writes:
AWS_REGION=<region>PL=$(aws ec2 describe-managed-prefix-lists \--filters Name=prefix-list-name,Values=com.amazonaws.$AWS_REGION.s3 \--query 'PrefixLists[0].PrefixListId' --output text)aws ec2 get-managed-prefix-list-entries --prefix-list-id "$PL" --query 'Entries[].Cidr' --output text
Save these as values.yaml. The chart pulls the image from the Marketplace registry in us-east-1 from any Region, through the endpoints above, so leave image.repository as it is.
deploymentId: <deployment-id> # stable forever; it's in the KMS conditions aboveimage:digest: sha256:<digest> # from the verification pageaws:region: <region>kmsKeyArn: <kms-key-arn>vault:url: postgresql://<aurora-writer-endpoint>:5432/tenet?sslmode=requirecredentialsSecret: { name: tenet-vault-db }lattice:gateway: { name: tenet-pii, namespace: lattice-system }detectInvokerRoleArns: [<role-of-the-app-that-calls-detect>]rehydrateInvokerRoleArns: [<role-allowed-to-rehydrate>]erasureInvokerRoleArns: [<role-allowed-to-erase-sessions>] # empty: nobody can eraseallowedSourceVpcIds: [<cluster-vpc-id>]erasureAuditS3Uri: s3://<audit-bucket>/erasure/detect:serviceAccount: { roleArn: <detect-role-arn> }rehydration:serviceAccount: { roleArn: <rehydrate-role-arn> }auditS3Uri: s3://<audit-bucket>/rehydrate/networkPolicy:awsEndpointCidrs: [<private-subnet-cidrs>] # subnets holding the VPC endpointsdatabaseCidrs: [<private-subnet-cidrs>] # subnets holding Auroras3PrefixListCidrs: [<from-the-command-above>]
helm upgrade --install tenet tenet-pii-enterprise-0.5.4.tgz -n tenet-pii -f values.yaml --wait --timeout 15mkubectl -n tenet-pii get podskubectl -n tenet-pii get httproute -o custom-columns='NAME:.metadata.name,DOMAIN:.metadata.annotations.application-networking\.k8s\.aws/lattice-assigned-domain-name'
The last command prints each service’s domain name, such as tenet-tenet-pii-enterprise-detect-0abc123.7d67968.vpc-lattice-svcs.us-east-1.on.aws. It can take a minute to appear. For detection only, add --set rehydration.enabled=false. You can then skip the rehydrate role and rehydrateInvokerRoleArns. Erasures are still audited, so keep the audit bucket.
To send VPC Lattice access logs to CloudWatch Logs, S3 or Firehose, set lattice.accessLogDestinationArn. After the first install, run kubectl -n tenet-pii get accesslogpolicy. If they show TargetNotFound, run the same helm upgrade again: the controller set them up before the services existed and doesn’t retry.
Set up a caller
Each application that calls Tenet, such as a pod with its own IAM role, needs four things:
| What | How |
|---|---|
| Its role in an invoker list | lattice.detectInvokerRoleArns, rehydrateInvokerRoleArns or erasureInvokerRoleArns. Each service’s auth policy names these roles per route and method. |
| An identity policy on that role | vpc-lattice-svcs:Invoke on the service, below. A caller in the same account needs both this and the auth policy; either alone is refused. |
| Its VPC allowed | In lattice.allowedSourceVpcIds, and associated with the Gateway’s service network. The controller associates the cluster’s VPC. |
| Signed requests | SigV4 for the service vpc-lattice-svcs, with the header x-amz-content-sha256: UNSIGNED-PAYLOAD, over HTTPS to the service’s domain name. |
The identity policy names the service’s ARN. The rehydrate one is the same, with tenet-tenet-pii-enterprise-rehydrate:
aws vpc-lattice list-services \--query "items[?name=='tenet-tenet-pii-enterprise-detect'].arn" --output text# then, on each detect caller's role:{"Version": "2012-10-17","Statement": [{"Effect": "Allow","Action": "vpc-lattice-svcs:Invoke","Resource": "<detect-service-arn>/*"}]}
Send your first document
The tenet-pii-lattice package signs requests with your AWS credentials and keeps one connection open, which matters: on VPC Lattice a new connection costs about 25 ms. It needs Python 3.11 or later and boto3. With no internet route, build it into your application’s image.
pip install https://platform.tenetlabs.com/enterprise/clients/tenet_pii_lattice-0.1.0.tar.gz
Call detect as a role listed in detectInvokerRoleArns, from a pod in the cluster:
DETECT_ENDPOINT=https://<detect domain name>echo "Follow up with Jane Smith (MRN 4482913) about her visit." | \python -m tenet_pii_lattice "$DETECT_ENDPOINT" deidentify
It returns the de-identified text and a session ID:
{"text": "Follow up with [FIRST_NAME_1] ([MRN_1]) about her visit.","session_id": "...", "expires_at": 1793000000.0, "findings": [...]}
Any SigV4 signer works in other languages, as long as it signs for vpc-lattice-svcs, sends x-amz-content-sha256: UNSIGNED-PAYLOAD and reuses connections. Routes are exact and case-sensitive. A request the policy doesn’t allow, or for any other path, gets a 403 or 404 from VPC Lattice and never reaches a pod. Calls to the Kubernetes Services directly are dropped by the network policies.
Rehydrate a response
Send text containing placeholders, such as your model’s answer, with the session ID from /deidentify, as a role listed in rehydrateInvokerRoleArns. Detect callers are refused.
REHYDRATE_ENDPOINT=https://<rehydrate domain name>echo "<text containing placeholders>" | \python -m tenet_pii_lattice "$REHYDRATE_ENDPOINT" reidentify <session_id>
The response holds the restored text, plus restored and unresolved for placeholders it did or didn’t find. Every call writes an audit record to your bucket naming the caller’s role session ARN, its account, the source VPC and the VPC Lattice request ID. If that write fails, nothing is restored. A session lasts one hour after its last /deidentify call, then returns 404. To change that, set TENET_PII_SERVE_VAULT_TTL in seconds through detect.extraEnv.
To require approval for each use, set lattice.requireApprovalTag to true. Every rehydration must then come from a role session carrying the tenet-approval-id session tag, which your approval workflow sets when it assumes the role, and VPC Lattice checks it on every call. Also put the approval ID in the role session name: the audit record names the session, so the approval appears in your audit trail.
Erase a session
To delete a session’s stored values before it expires, call DELETE /sessions/<session_id> on the detect service as a role listed in erasureInvokerRoleArns. It returns 204, including for an ID that doesn’t exist. Afterwards, rehydrating or de-identifying with that ID returns 404. Each erasure is audited before it runs, and an erasure that can’t be recorded isn’t performed.
python -m tenet_pii_lattice "$DETECT_ENDPOINT" erase <session_id>
Billing
AWS bills usage through your AWS account, at the prices on your Marketplace offer. Each detect pod reports its hourly quantities to AWS Marketplace metering over the VPC endpoint, including when it stops. Rehydration is not billed. VPC Lattice itself is billed by AWS, as usual for your account.
Sending hourly usage counts to Tenet is off by default. If you turn it on with telemetry.egress.enabled and a Secret holding a Tenet API key, the chart lets the detect pods reach public addresses on port 443. Your node subnets need a route to a NAT gateway for it. Those records hold counts only, never text or values.
What to expect
- Facility names: these aren’t reliably redacted, so validate results on your own data.
- A first install: rehydrate pods wait up to two minutes for detect to create the first key, and may log “vault not ready yet” while they do. Pods also wait for the controller to create their VPC Lattice service. That’s expected.
- Version numbers: /healthz and audit records report the detection engine’s version, such as 0.10.0. The product release is 0.5.4.
- Pods that never become ready: check that the HTTPRoutes have a domain name. If not, the controller’s logs usually name a missing endpoint or permission.
Remove the release
Run helm uninstall tenet -n tenet-pii. This keeps your KMS key, vault and audit bucket. In COMPLIANCE mode, nobody can delete audit objects until their retention ends. In GOVERNANCE mode, an administrator can delete them with --bypass-governance-retention.
To remove everything after a trial, delete in this order: the Helm release and namespaces, the Gateway, the controller, the node group and cluster, the Aurora database, the audit bucket, the VPC endpoints and the VPC, then the IAM roles. Schedule the KMS key for deletion last.