new | show | ask | jobs Github

gavinray 6 hours ago

If this actually works, this is such a huge boon. Let me share why:

I work on connectors to various datasources, mostly SQL DB's

The most useful tests are the integration tests, sending actual queries to the source

For most DB's we have a giant Docker Compose which runs a local instance

But things like Snowflake, Databricks, Redshift, Athena suck to test because we need an external hosted instance

I would kill to be able to run test versions of all of these locally

▲

skeeter2020 4 hours ago | parent | next [-]

Why are you writing tests on the actual provider? IME these types of tests are slow, brittle and don't give you much value. Even if you get a true positive, what can you do about it?

	▲	gavinray 2 hours ago \| parent \| next [-]
		Because those tests are the only ones that prove the connector is working as intended Our connectors receive a generic query IR from a Datafusion-based engine Each connector is then responsible for faithfully translating the query IR into the semantically equivalent query for the underlying data source For a SQL DB this would be a SQL query, for MongoDB a JSON predicate query, for csv/json files it would be an in-memory predicate engine, etc Running the generated query against the target data source and asserting that this doesn't throw an error and that the records returned are both correct and in the right order is the true test
	▲	viccis 11 minutes ago \| parent \| prev \| next [-]
		When it comes to platforms like Databricks, integration tests are honestly the only useful ones. Grug remains vindicated as always.
	▲	dionian 3 hours ago \| parent \| prev [-]
		I assume integration tests.

▲

tomnipotent 2 hours ago | parent | prev | next [-]

For cloud databases, in your use case do you care that the queries actually execute or only that they parse? I with every cloud service had their own version of LocalStack/Moto, and while not perfect they've really helped to improve test coverage.

	▲	gavinray an hour ago \| parent [-]
		Unfortunately they have to actually execute and have identical semantics to the target source I recall seeing a project based on DuckDB parser and engine which emulated dialects for one of these but because the sortinh and other behavior is different it wasn't usable

▲

efromvt 6 hours ago | parent | prev [-]

[dead]