| ▲ | MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark(dmux.github.io) |
| 83 points by kermatt 2 days ago | 17 comments |
| |
|
| ▲ | gavinray 5 hours ago | parent | next [-] |
| If this actually works, this is such a huge boon. Let me share why: I work on connectors to various datasources, mostly SQL DB's The most useful tests are the integration tests, sending actual queries to the source For most DB's we have a giant Docker Compose which runs a local instance But things like Snowflake, Databricks, Redshift, Athena suck to test because we need an external hosted instance I would kill to be able to run test versions of all of these locally |
| |
| ▲ | skeeter2020 3 hours ago | parent | next [-] | | Why are you writing tests on the actual provider? IME these types of tests are slow, brittle and don't give you much value. Even if you get a true positive, what can you do about it? | | |
| ▲ | gavinray an hour ago | parent | next [-] | | Because those tests are the only ones that prove the connector is working as intended Our connectors receive a generic query IR from a Datafusion-based engine Each connector is then responsible for faithfully translating the query IR into the semantically equivalent query for the underlying data source For a SQL DB this would be a SQL query, for MongoDB a JSON predicate query, for csv/json files it would be an in-memory predicate engine, etc Running the generated query against the target data source and asserting that this doesn't throw an error and that the records returned are both correct and in the right order is the true test | |
| ▲ | dionian 2 hours ago | parent | prev [-] | | I assume integration tests. |
| |
| ▲ | tomnipotent an hour ago | parent | prev [-] | | For cloud databases, in your use case do you care that the queries actually execute or only that they parse? I with every cloud service had their own version of LocalStack/Moto, and while not perfect they've really helped to improve test coverage. | | |
| ▲ | gavinray 26 minutes ago | parent [-] | | Unfortunately they have to actually execute and have identical semantics to the target source I recall seeing a project based on DuckDB parser and engine which emulated dialects for one of these but because the sortinh and other behavior is different it wasn't usable |
|
|
|
| ▲ | akbarian 5 hours ago | parent | prev | next [-] |
| Getting tired of the standard AI design, all websites look the same now |
| |
| ▲ | smashed 4 hours ago | parent | next [-] | | I would say go straight to their GitHub but there again, same thing. With a list of "Known gaps" and implemented engines listed with status like "Real", and of course the Quick start first, as Claude loves to do. It's not "wrong", it just shows low effort. But then again it's just something someone needed and shared it for free so let's not complain too much. All the products and saas launching with obvious AI landing pages, that the authors did not even care to trim down, that's another story. | |
| ▲ | pjmlp 5 hours ago | parent | prev [-] | | Before it was all those common template sites. |
|
|
| ▲ | lysecret 3 hours ago | parent | prev | next [-] |
| Want to hijack this someone made experience recently with https://ducklake.select/ ? We need to setup a warehouse soon and this looks enticing. |
| |
| ▲ | denom an hour ago | parent | next [-] | | Ducklake works well. So far it is still under active development (post AWS acquisition). There is support for local disk-based and remote S3 based datalakes, so integration tests become really easy. Seems stable and all the bugs I've found have already been patched. | |
| ▲ | Zizizizz 2 hours ago | parent | prev [-] | | I'd personally love to try this in earnest as well |
|
|
| ▲ | asabla 2 hours ago | parent | prev | next [-] |
| Oh wow! This would've been so useful at my previous work place. I'll save this one for later. Looks pretty decent too for homelabs! I wonder if there is any plans on extending it further? (just a bit) |
|
| ▲ | ngruhn 6 hours ago | parent | prev | next [-] |
| You should call it "pond". |
|
| ▲ | isoprophlex 6 hours ago | parent | prev | next [-] |
| Great for CI i guess; or is this meant as a cost savings measure where you can swap out your databricks for self-hosted this? |
|
| ▲ | dominotw 3 hours ago | parent | prev [-] |
| Wtf is databricks now anyways? Looks like their ceo is hawking cheap chinese models and some silly harness called ominiagent. Anyone use ominiagent?
I thought they were building their own models at somepoint? what happened to that. Also looks like they did one full round of english alphabet and are on series L second time around. I feel like we might have some a tech theranos on our hands here. |
| |
