Friday, April 12, 2024

Databricks AWS private link with conditional DNS forwarder

Databricks allows workspace to be accessible via a private ip so not publicly available.
This is useful in some cases where you want to restrict users to used it only if are connected to a VPN or equivalent.
These are the steps to achive this goal
  1. Create a custom VPC in AWS
  2. Connect the VPC to your infrastructure (VPN/Direct Connect)
  3. Create VPC Endpoint for frontend and backend
  4. Register private link in the databricks account and associate it to your workspace
  5. Change access mode to Private from Public in the Private Access Settings
  6. Create a private zone in Route53
  7. Create an inbound resolver in Route53
  8. Add an A entry in the zone that points to the address of the VPC Endpoint (step3)
  9. Add a forwarder in your Private DNS to point to the inbound resolver (step 7) ip address(es)
For more details Databricks documentation explains in more details docs.databricks.com (search for private link). Once all is in place the flow is
+---------+                               +-------------+                                            +-----------+ +-------------------------+                                         +-------------+
| Client  |                               | PrivateDns  |                                            | PublicDns | | Route53InboundResolver  |                                         | Route53Zone |
+---------+                               +-------------+                                            +-----------+ +-------------------------+                                         +-------------+
     |                                           |                                                         |                    |                                                             |
     | my-workspace.cloud.databricks.com         |                                                         |                    |                                                             |
     |------------------------------------------>|                                                         |                    |                                                             |
     |                                           |                                                         |                    |                                                             |
     |                                           | my-workspace.cloud.databricks.com                       |                    |                                                             |
     |                                           |-------------------------------------------------------->|                    |                                                             |
     |                                           |                                                         |                    |                                                             |
     |                                           |       CNAME nvirginia.privatelink.cloud.databricks.com. |                    |                                                             |
     |                                           |<--------------------------------------------------------|                    |                                                             |
     |                                           |                                                         |                    |                                                             |
     |                                           | nvirginia.privatelink.cloud.databricks.com ?            |                    |                                                             |
     |                                           |----------------------------------------------------------------------------->|                                                             |
     |                                           |                                                         |                    |                                                             |
     |                                           |                                                         |                    | nvirginia.privatelink.cloud.databricks.com ?                |
     |                                           |                                                         |                    |------------------------------------------------------------>|
     |                                           |                                                         |                    |                                                             |
     |                                           |                                                         |                    |                                      Address is 172.16.0.10 |
     |                                           |                                                         |                    |<------------------------------------------------------------|
     |                                           |                                                         |                    |-----------------------------------------------------------\ |
     |                                           |                                                         |                    || nvirginia.privatelink.cloud.databricks.com A 172.16.0.10 |-|
     |                                           |                                                         |                    ||----------------------------------------------------------| |
     |                                           |                                                         |                    |                                                             |
     |                                           |                                                Answer address is 172.16.0.10 |                                                             |
     |                                           |<-----------------------------------------------------------------------------|                                                             |
     |                                           |                                                         |                    |                                                             |
     |                    Connect to 172.16.0.10 |                                                         |                    |                                                             |
     |<------------------------------------------|                                                         |                    |                                                             |
     |                                           |                                                         |                    |                                                             |