Data mesh on AWS with SageMaker Unified Studio: what I take away from practice

Data mesh sounds like organizational theory: domains own their data, data is treated as a product, a platform enables self-service and governance is federated. On slides that is quickly explained. In an AWS account, what is left of it is a handful of very concrete things: a Glue table, a Lake Formation permission, an entry in a catalog and someone who approves a subscription.

For some time now I have been looking after the data side of a platform whose figures several departments need in their own AWS accounts. It started with Amazon DataZone; today it runs on Amazon SageMaker Unified Studio, SMUS for short, which is built on DataZone. This article is not a tour of every feature. It describes what actually matters day to day, where things fail silently and what I would do differently from the start next time.

The short version first: SMUS is a good shop window and a good approval process. The actual data mesh sits underneath, in Glue and Lake Formation, and that is where it is decided whether it works.

Why a data mesh at all

Many will know the starting point: revenue and usage data sits in S3 and is analysed with Athena. Other teams want the same numbers, so they get a CSV export, a bucket policy or an IAM role with read access to a prefix. Two years later nobody knows exactly who reads which data, in which format, and whether they rely on it.

The problem is rarely the technology. The problem is that nobody can answer a question everyone asks sooner or later: if I change this column, who will notice? A data mesh answers it through structure. Whoever provides data publishes it as a data product in a catalog. Whoever wants to use it subscribes there. The approval is documented, the permission follows from it automatically, and the producer can see who has subscribed.

What data mesh means on AWS in concrete terms

The first hurdle is linguistic, and it cost me more time than I would like to admit. In the data mesh literature a domain is a business area with its own responsibility for data, such as "sales" or "logistics". In DataZone and SMUS, however, a domain is the whole frame: one catalog, one portal, one user directory, often a single one for the entire company. What counts as a domain in the data mesh sense corresponds more closely to a project or a domain unit in SMUS.

If you do not settle this early, you end up in conversations where both sides say "domain" and mean different things. I now write the mapping down:

The flow is then easy to describe. The producing project has a data source that regularly reads a Glue database and imports every matching table into the catalog as an asset. The asset is published. A consuming project finds it and requests a subscription, the producer approves, and SMUS creates the required Lake Formation permissions for the role of the consuming project. From then on the consumer queries the table with Athena without anyone having touched a bucket policy.

So much for the theory. The following sections are about what can go wrong at each of these stations.

A database of its own as the shop window

Fortunately I made the most important decision early: whatever is shared lives in a Glue database of its own. The application's internal tables live in one database, the data products in another. The second is the shop window, and only what is in it can become an asset at all.

That sounds like tidiness, but it has two tangible advantages. First, an internal table cannot end up in the catalog by accident because a data source filters too generously. Second, permissions can be granted across the whole database. Our cross-account grants are defined in Terraform as a wildcard over all tables of the shop window database:

resource "aws_lakeformation_permissions" "consumer_tables" {
  for_each  = toset(var.consumer_account_ids)
  principal = each.value

  permissions                   = ["SELECT", "DESCRIBE"]
  permissions_with_grant_option = ["SELECT", "DESCRIBE"]

  table {
    database_name = aws_glue_catalog_database.products.name
    wildcard      = true
  }
}

A new data product therefore needs not a single additional permission. The table is created, inherits the grants, and the rest happens through the catalog. The with_grant_option is not a detail: only with it can the consumer account pass the rights on to its own roles, that is, to the role of its SMUS project.

You cut a data product by its path

The first data product I built for another team was the figures of a single business partner. The obvious solution would have been to share the existing table with the figures of all partners and ask the consumer to filter by partner. That would have been quick, and wrong.

The existing table was partitioned by year, month and day. The partner sat below the day in the path and was just an ordinary column as far as Glue was concerned. A Glue table can only point at a prefix, though, not at a pattern somewhere in the middle. Sharing it means sharing every partner, and every subscriber could have read the figures of all the others.

Lake Formation offers data filters on rows and columns for such cases. I still decided against them, because there was a simpler solution: the application already wrote the same data a second time for an export, and there the partner came before the date in the path:

s3://…/partner=a/year=2026/month=10/day=15/…

A table that points at partner=a/ cannot contain anything but this partner. No view, no filter, no row rule that someone later relaxes by mistake. A data product that cannot contain the wrong data is easier to verify than one that filters it out. It is the same thinking as with permissions in general: I prefer a boundary that holds by construction to one that holds by configuration.

So when you plan your first data product, look at the path structure in S3 first. It decides how finely you can cut, and reworking the partitioning is usually more expensive than a second, suitably cut copy.

Lake Formation is the actual mesh

In the SMUS portal a subscription looks like a single click. Underneath sits Lake Formation, and that is where almost every problem that cost me time was. I would pass three of them on to anyone.

Hybrid access mode needs an opt-in

If you put an existing data lake under Lake Formation after the fact, you usually choose hybrid access mode. The existing IAM permissions keep working and Lake Formation is added on top. That is exactly right for a migration, but it has a peculiarity: Lake Formation permissions only take effect for a principal once it has been opted in for the resource. Without that opt-in, a grant exists, is visible and correct, and does nothing.

In practice it looks like this: the subscription has been approved, SMUS reports success, and the consumer still gets no data in Athena. No error message pointing to the missing opt-in. That is why ours sits next to the grants in Terraform, for the database and for all tables:

resource "aws_lakeformation_opt_in" "consumer_tables" {
  for_each = toset(var.consumer_account_ids)

  principal {
    data_lake_principal_identifier = each.value
  }

  resource_data {
    table {
      catalog_id    = data.aws_caller_identity.current.account_id
      database_name = aws_glue_catalog_database.products.name
      wildcard      = true
    }
  }

  depends_on = [aws_lakeformation_opt_in.consumer_database]
}

Two details are hidden in there. The catalog_id is set explicitly, because otherwise the API fills it in itself and Terraform wants to replace the resource on every plan. And the depends_on serializes the opt-ins for database and tables, because the API rejects concurrent modifications with a ConcurrentModificationException.

The list of administrators is complete or dangerous

To set opt-ins and grants through Terraform, the deploy role has to be a data lake administrator. IAM rights such as lakeformation:* are not enough; that is a separate layer. The Terraform resource for it, aws_lakeformation_data_lake_settings, is authoritative, though: its list of admins replaces the existing one entirely.

If you only enter your own deploy role, the next apply strips admin rights from everyone else. That includes the SSO roles of the humans and the service roles that DataZone and SMUS created themselves. After that, publishing stops working first, and then nobody can get into the console to fix it.

So before taking it over I read the current list with aws lakeformation get-data-lake-settings, import the resource and let the plan fail on a condition if the list is empty in production or contains a placeholder:

lifecycle {
  precondition {
    condition = (
      var.environment != "prod" || length(var.lf_admins) > 0
    ) && alltrue([
      for a in var.lf_admins :
      can(regex("^arn:aws:iam::[0-9]{12}:(role|user)/", a))
    ])
    error_message = "The admin list is incomplete."
  }
}

A comment in the code would have served the same purpose, but comments do not stop anyone. A condition in the plan does.

Taking admin rights away again

The flip side: over the years, admins accumulate. In our case an old deploy role that had long since been doing something else still had data lake admin. Before removing it, I checked in CloudTrail which calls its sessions actually made. They were exclusively calls for container deployments, not a single Lake Formation call. That turned the decision from a guess into a fact. I describe the approach behind it in more detail in cleaning up IAM in a grown AWS account.

The data source decides what becomes visible

A Glue table is not yet an asset. It becomes one when a data source of the project picks it up on its next run. Every data source has a table filter, and whatever does not match never shows up in the catalog. Again without an error message.

In our case the data sources of the producing project had originally been created by hand in DataZone, not in Terraform. After the switch they could no longer be edited in the portal. What remained was a single source that still ran daily against the shop window database, and its filter only matched tables with a particular prefix that came from the first use case.

The new table did not have that prefix and therefore never became an asset. The pragmatic fix was to rename it so that it matched the filter. Since then the prefix has been purely a routing device and says nothing about the content, which I wrote explicitly into the Terraform code. Otherwise the next reader takes it for a mistake.

Even the rename was not trivial in Terraform. Renaming a Glue table means replacing it, and our pipeline does not let through any plan that deletes such a resource. So I created the new table as a new resource and only released the old one from state:

removed {
  from = aws_glue_catalog_table.partner_figures
  lifecycle {
    destroy = false
  }
}

The orphaned Glue entry had no subscribers and was removed by hand afterwards. The real lesson is a different one, though: data sources belong in code from day one, and their filter should describe a convention, not the origin of the first dataset. A prefix such as product__ would never have let the problem arise. Why I usually describe such resources with Terraform is covered in Terraform or AWS CDK?

A data product is a contract

As soon as someone has subscribed to a table, its schema is a contract. That sounds obvious and is broken all the time anyway, because a Glue table does not feel like an API.

An example from practice: the data is stored as CSV, and Glue reads CSV with the LazySimpleSerDe. It maps columns by position, not by the name in the header line. The column order of the table therefore has to match the order in which the application writes exactly, including an outdated column that nobody is supposed to read any more but that is still in the file. Anyone who "tidies" it out of the table shifts every following value by one column. The query keeps running, the numbers are wrong.

I therefore treat data products by three rules:

• Every column has a comment that is visible in the catalog: unit, currency, meaning of special values. If it says what an empty value means, nobody needs to ask.
• Extend, yes; rebuild, no. When the same figures were also needed in a finer breakdown, I did not rebuild the existing table, which another consumer was already using. There is now a second data product with its own table. The existing one stayed unchanged.
• The producer reads along. A monitor on the freshness of the data belongs to the product, not to the consumer. Whoever offers a data product should notice before their subscribers that nothing arrived yesterday.

Retiring is part of it

A catalog grows. Sooner or later there are assets that have been replaced by better ones, and tables that only still exist because nobody is sure whether they are needed. This is where an advantage of the catalog shows that is easy to overlook while building it: it knows who has subscribed.

Without a catalog, retiring a table starts with detective work: reading access logs, asking teams, waiting to see who complains. With a catalog it starts with a look at the subscriptions. Only once nobody is listed there do the asset, the Glue table and finally the data in S3 go away, in that order. That way even a data product made redundant by a better one can be retired without anything breaking.

When it pays off and when it does not

SMUS is not a small tool. It brings a domain, projects, service roles, a portal and a lot of terminology, and in some places you notice that the product is still maturing: data sources that can no longer be edited after a migration, failures without any message, and Terraform coverage that lags behind the console.

So I would not recommend it for every data share:

The last point is the most important one. A data mesh without producers who really own their data is an expensive catalog full of stale entries. SMUS makes ownership visible; it does not create it.

Where I would start today

If I built the whole thing again, I would go in this order:

1. A Glue database of its own for data products, separate from all internal tables.
2. Lake Formation as code: the admin list fully imported and guarded, S3 paths registered, grants and opt-ins as a wildcard over the product database.
3. Data sources in Terraform, with a filter that describes a naming convention and not the first use case.
4. Cut the first data product by its S3 path, not by a filter, and comment every column.
5. Test the whole route once with a real consumer account: publish, subscribe, approve, query in Athena. Only when rows come back does it work.
6. Monitor freshness, at the producer, with an alarm that reaches someone.

None of this is spectacular. But each of these steps stands for a failure that went through without a message in our case. A data mesh on AWS rarely fails loudly. It fails because a table does not show up in the catalog, a subscription returns no data or a column shifts, and nobody notices. If you capture these silent spots in code and checks from the start, SMUS gives you a tool that genuinely makes sharing data easier.