| ▲ | Show HN: Streaming gigabyte medical images from S3 without downloading them(github.com) |
| 84 points by el_pa_b 7 hours ago | 22 comments |
| |
|
| ▲ | Sleaker 3 minutes ago | parent | next [-] |
| Maybe a bit pedantic, but if you're streaming it, then you're still downloading portions of it, yah? Just not persisting the whole thing locally before viewing it. Edit: Looks like this is a slight discrepancy between the HN title and the GitHub description. |
|
| ▲ | matthberg 6 hours ago | parent | prev | next [-] |
| Seems very similar to how maps work on the web these days, in particular protomap files [0]. I wonder if you could view the medical images in leaflet or another frontend map library with the addition of a shim layer? Cool work! 0: https://protomaps.com/ |
| |
| ▲ | el_pa_b 6 hours ago | parent [-] | | Thanks! Indeed, digital pathology, satellite imaging and geospatial data share a lot of computational problems: efficient storage, fast spatial retrieval/indexing. I think this could be doable. As for digital pathology, the field is very much tied to scanner-vendor proprietary formats (SVS, NDPI, MRXS, etc). |
|
|
| ▲ | rwmj 5 hours ago | parent | prev | next [-] |
| https://dicom.nema.org/dicom/dicomwsi/ Interesting guide to the Whole Slide Images (WSI) format. The surprising thing for me is that compression is used, and they note does not affect use in diagnostics. Back in the day we used TIFF for a similar application (X-ray detector images). |
| |
| ▲ | yread an hour ago | parent [-] | | Digital pathology are just a lot bigger than radiology, we regularly see slides 500k x 500k pixels. |
|
|
| ▲ | tokyovigilante 5 hours ago | parent | prev | next [-] |
| This is really a job for JPEG-XL, which supports decode of portions of larger images and has recently been added to the DICOM standard. |
| |
|
| ▲ | yread an hour ago | parent | prev | next [-] |
| You could probably do it completely clientside. I have a parser for 12 scanner formats in js. It doesnt read the pixels, just parses metadata but jpeg is easy and most common anyway |
|
| ▲ | lametti 5 hours ago | parent | prev | next [-] |
| Interesting - I'm not so familiar with S3 but I wonder if this would work for WSI stored on-premises. Imposing lower network requirememts and a lightweight web viewer is very advantageous in this use case. I'll have to try it out! |
| |
| ▲ | el_pa_b 5 hours ago | parent [-] | | When WSI are stored on-premise, they are typically stored on hard drives with a filesystem. If you have a filesystem, you can use OpenSlide, and use a viewer like OpenSeaDragon to visualize the slide. WSIStreamer is relevant for storage systems without a filesystem. In this case, OpenSlide cannot work (it needs to seek and open the file). |
|
|
| ▲ | invaderJ1m 2 hours ago | parent | prev | next [-] |
| How does this compare to things like COGs (Cloud Optimised GeoTIFFs) or other binary blob + index raster pyramid formats? Was there a requirement to work with these formats directly without converting? |
| |
| ▲ | el_pa_b 35 minutes ago | parent [-] | | Yes there is a requirement to work with the vendor format. For instance, TCGA (The Cancer Genome Atlas - a large dataset of 12k+ human tumor cases) has mostly .svs files (scanned with an Aperio scanner). We tend to work with these formats as they contain all the metadata we need. Sometimes, it happens that we re-write the image in a pyramidal TIFF format (happened to me a few times, where NDPI images had only the highest resolution level, no pyramid), in which case COGs could work. |
|
|
| ▲ | Nora23 5 hours ago | parent | prev | next [-] |
| How does this handle images with different compression formats? |
| |
| ▲ | el_pa_b 32 minutes ago | parent [-] | | Currently we only support TIFF and SVS with JPEG and JPEG2000 compression formats. I plan on supporting more file extensions (e.g. NDPI, MRXS) in the future, each with their own compression formats. |
|
|
| ▲ | andrewstuart an hour ago | parent | prev | next [-] |
| Please don’t use AWS S3 there’s vast numbers of much cheaper compatible choices. |
| |
| ▲ | lijok 13 minutes ago | parent | next [-] | | I guess by "compatible" you mean the data plane. There are choices that speak the S3 data plane API (GetObject, ListBucket, etc). There are no alternatives that support most of the AWS S3 functionality such as replication, event notifications. | |
| ▲ | el_pa_b 34 minutes ago | parent | prev | next [-] | | As data scientists, we usually don't get to choose. It's usually up to the hospital or digital lab's CISO to decide where the digitized slides are stored, and S3 is a fairly common option. That being said, I plan to support more cloud platforms in the future, starting with GCP. | |
| ▲ | kube-system 11 minutes ago | parent | prev | next [-] | | “Cheap” is not always the #1 requirement for a project. | |
| ▲ | thenaturalist an hour ago | parent | prev [-] | | Pretty bold half claim while not backing it up with a single data point. :D |
|
|
| ▲ | tonyhart7 5 hours ago | parent | prev [-] |
| hey, I need this |
