Does anyone know how access control works to the underlying s3 objects? I didn’t see anything regarding grants in the docs.

▲

pgguru 2 days ago | parent | next [-]

Hi, one of the developers here. You define credentials that can access the S3 buckets and use those as DuckDB secrets, usually in an init script for pgduck_server. (You can see some examples of this in the testing framework.)

I'll see if we can improve the docs or highlight that part better, if it is already documented—we did move some things around prior to release.

▲

onderkalaci 2 days ago | parent | next [-]

Maybe this could help: https://github.com/Snowflake-Labs/pg_lake?tab=readme-ov-file...

▲

mberning 2 days ago | parent | prev [-]

Interesting. I am working on a project to integrate access management to iceberg/parquet files for sagemaker. Controlling what users logged into sagemaker studio have access to in s3. It’s fine using static policies for mvp, but eventually it needs to be dynamic and integrated into enterprise iam tools. Those tools generally have great support for managing sql grants. Not so much for s3 bucket policies.

	▲	pgguru 2 days ago \| parent [-]
		DuckDB secrets management supports custom IAM roles and the like; at this point we are basically treating the pgduck_server external system as a black box. For the postgres grants themselves, we provide privs to allow read/write to the remote tables, which is done via granting the `pg_lake_read`, `pg_lake_write` or `pg_lake_read_write` grants. This is a blanket all-or-nothing grant, however, so would need some design work/patching to support per-relation grants, say. (You could probably get away with making roles in postgres that have the appropriate read/write grant, then only granting those specific roles to a given relation, so it's probably doable though a little clunky at the moment.)

▲

mslot 2 days ago | parent | prev [-]

There are Postgres roles for read/write access to the S3 object that DuckDB has access to. Those roles can create tables from specific files or at specific locations, and can then assign more fine-grained privileges to other Postgres roles (e.g. read access on a specific view or table).