Remix.run Logo
incomplete 12 hours ago

yep, i work for a major university and our canvas instance is down. this is really, really bad.

edit: here's the list of impacted universities (unsure if they all have their canvas instances offline, but i'd be surprised if not): http://91.215.85.103/pay_or_leak/instructure_affected_school...

starkrights 10 hours ago | parent | next [-]

The source txtfile has since either been dos'd or deleted (at least it was when I tried to access)

Someone dumped the content into a google doc on reddit[1] if anyone's interested.

[1]: https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sT...

rigrassm 7 hours ago | parent [-]

> The source txtfile has since either been dos'd or deleted (at least it was when I tried to access)

> Someone dumped the content into a google doc on reddit[1] if anyone's interested.

> [1]: https://docs.google.com/document/d/1MTktVSwTUM5I_w7bKNGj94sT...

Thanks for linking this. Ended up finding my kids school district on the list unfortunately.

12_throw_away 12 hours ago | parent | prev | next [-]

tbh this has me wondering if canvas "instances" are actually as isolated and segregated from each other as they're supposed to be.

javawizard 10 hours ago | parent | next [-]

Define "as they're supposed to be".

Back when I worked for Instructure ~10 years ago, Canvas was effectively a single, giant, monolithic multitenant app with one instance backed by several thousand app servers and ~100 separate Postgres database clusters that any app server could talk to.

Schools were grouped onto pools of app severs and Postgres database clusters more or less according to locality and cluster availability. I want to say a handful of the largest schools got their own clusters, but I'm not certain, and at any rate their clusters could certainly all talk to each other.

It was actually kind of neat from a technical perspective: any Rails model across the entire Canvas world could have a "foreign key" pointing to any other Rails model anywhere else. Among other things, this allowed for users who could administer multiple Canvas organizations, even if those organizations resided on different Postgres clusters. https://github.com/instructure/switchman is their gem that made that all work. (I put "foreign key" in quotes because the whole thing was implemented in software, not with actual database FKs, for obvious reasons.)

---

Of course, the massive downside to that sort of thing is that if you manage to pop one Canvas app server, you have the keys to the kingdom. I wonder if they'll sharpen the edges between clusters in response to this...

---

(Disclaimer: I left Instructure back in 2017; much could have changed since then, and my memory could be faulty about the specifics. Caveat emptor.)

wky 11 hours ago | parent | prev | next [-]

It's possible that Instructure's servers got compromised:

dig canvas.ucdavis.edu

    [...]
    
    ;; ANSWER SECTION:
    canvas.ucdavis.edu. 1974 IN CNAME ucdavis-vanity.instructure.com.
    ucdavis-vanity.instructure.com. 60 IN A 18.173.121.125
    ucdavis-vanity.instructure.com. 60 IN A 18.173.121.103
    ucdavis-vanity.instructure.com. 60 IN A 18.173.121.15
    ucdavis-vanity.instructure.com. 60 IN A 18.173.121.18
dig canvas.duke.edu

    ;; ANSWER SECTION:
    canvas.duke.edu. 300 IN CNAME duke-vanity.instructure.com.
    duke-vanity.instructure.com. 60 IN A 18.173.121.125
    duke-vanity.instructure.com. 60 IN A 18.173.121.18
    duke-vanity.instructure.com. 60 IN A 18.173.121.103
    duke-vanity.instructure.com. 60 IN A 18.173.121.15
mrsvanwinkle 11 hours ago | parent [-]

that's what the screenshot says. They rooted Instructure servers.

SamuelAdams 10 hours ago | parent | prev [-]

It depends on what you pay for. If you need FedRamp or IL4+ compliance you are likely on dedicated infrastructure. Everyone else uses multi tenancy.

GaryBluto 5 hours ago | parent | prev | next [-]

https://web.archive.org/web/20260507042014fw_/http://91.215....

Cider9986 9 hours ago | parent | prev | next [-]

Here's an archive https://archive.is/eB2hE

mrsvanwinkle 12 hours ago | parent | prev [-]

[dead]