| ▲ | blahedo 8 hours ago |
| Perspective from the trenches: I teach at a university that uses Canvas. We are in our final exams period right now. We got our first email (from Academic Affairs) notifying us that it was down at 5:17pm EDT this afternoon, with little info; followup emails were sent at 6:24 and 6:57 with more info, but mostly about how we would be compensating for it and not about what actually was going on (other than, "nationwide shutdown" and "cybersecurity attacks", no further detail). I don't get a sense that they know much more than that, not that I would expect them to. A perhaps telling detail: they're instructing us to have students email us directly with any work that had been submitted via Canvas. That suggests that they have no particular confidence that it will come back up soon. I personally am only slightly affected; as a CS professor a lot of my students' work is done on department machines, and submitted that way, and I do the actual exams on paper. More importantly, I've never liked or trusted Canvas's gradebook, and so although I do upload grades to Canvas so students can see them, my primary gradebook is always a spreadsheet I maintain locally. But I have a lot of colleagues for whom this is catastrophic at a level of "the whole building burnt down with all my exams and gradebooks in it"---even many of those that teach 100% in person have shifted much or all of their assessment into Canvas (using the Canvas "quiz" feature for everything up to and including final exams), and use the Canvas gradebook as their source-of-truth record. We've been encouraged to do so by our administration ("it makes submitting grades easier"). For faculty in that situation, they have few or zero artifacts that the students have produced, the students themselves don't have the artifacts to resubmit via email because they were done in Canvas in the first place, and they have no record of student grades or even attendance (because they managed that all inside Canvas). I guess they have access to the advisory midterm grades from March, if they submitted them (most do, some don't), but that might be it. My gut feeling on this is that this is either resolved in hours (they have airgapped backups and can be working as soon as they can spin up new servers), or weeks (they don't). Very little in-between. And if that's true and we wake up tomorrow with this unresolved, I really have no idea what a lot of professors at my university and across the country are going to do to submit grades that are fair and reasonable. In the extreme case, they may have to revert to something we did in the pandemic semester (and before that, at my school, in the semester that two major academic buildings actually did burn to the ground a week before finals): let classes that normally count for a grade just submit grades as pass-fail. Because what else can you do? (Well, one thing you can do is not put your eggs all in one basket, and not trust "the cloud" quite so much, but that ship's already sailed. I do wonder if in the longer term, anybody learns any lessons from this....) UPDATE: As of 11:45pm EDT, my university's canvas instance is up and running! Here's hoping it stays (but I'll be downloading some stuff just in case...) |
|
| ▲ | JumpCrisscross 7 hours ago | parent | next [-] |
| > the students themselves don't have the artifacts to resubmit via email because they were done in Canvas It’s so simple to send an e-mail to the student with relevant records on completion of a quiz or whatnot. They don’t do it, because they want to control the data. (And universities don’t insist on it for who knows what reason.) |
| |
| ▲ | gucci-on-fleek 5 hours ago | parent | next [-] | | I've never used Canvas before, but all the LMSes that I've used allow students to enable emails whenever anything is updated, including when grades are posted. This is off by default because it's often 10+ emails a day, because many teachers post notes once a day, and with 5 classes, that adds up pretty quick. I personally have it enabled because it's pretty manageable with some custom Outlook rules, but setting this up is well beyond the capabilities of most students. | | |
| ▲ | dotancohen 4 hours ago | parent [-] | | > setting this up is well beyond the capabilities of most students.
Setting up custom email filters is beyond the capabilities of most students? What are they learning? Where will they be qualified to work? | | |
| ▲ | weird-eye-issue 4 hours ago | parent | next [-] | | Most graduates aren't really qualified to work anywhere that they couldn't have worked before going to college in the first place. | | |
| ▲ | smcin 3 hours ago | parent [-] | | You mean graduates of US colleges? Not colleges in general. Or non-technical graduates of US colleges? |
| |
| ▲ | metaengies 3 hours ago | parent | prev | next [-] | | > Where will they be qualified to work? Going by a certain story 2 years ago, their concern should be that they're overqualified for Meta. It doesn't help that gmail, which is the only serious direct competition to outlook, straight up doesn't do "folders" and instead goes with markers. So you can't really just put a filter that drags all the 100 low-priority alerts in what would count as a first degree abstraction of "place where things are sorted into". No, there are two layers of abstraction between point A and B of things, sorter and sorted things. The result? Muggles can't recognize the heck you're describing and refuse to even acknowledge the possibility. | | |
| ▲ | user_7832 29 minutes ago | parent | next [-] | | > It doesn't help that gmail, which is the only serious direct competition to outlook, straight up doesn't do "folders" and instead goes with markers. While true, unless I'm mistaken, markers (I assume you're referring to tags) can be nested to provide a pseudo-folder hierarchy, and with proper filters you can remove the "inbox" tag and have the mail only show up under the specific tag. TBH I don't fully mind it, it lets you classify an email in multiple ways (eg "See Later" as well as "Work related"). | |
| ▲ | GTP an hour ago | parent | prev | next [-] | | I partially solve this by using Thunderbird on my laptop. When I get emails on my smartphone (on the Gmail app), they unfortunately all go to the inbox. But the moment I open Thunderbird, it nicely organizes them for me. | | |
| ▲ | dotancohen an hour ago | parent [-] | | I use Thunderbird on both the desktop and Android. Love it. Perhaps Outlook is difficult to configure. Thunderbird is intuitive. |
| |
| ▲ | teiferer 2 hours ago | parent | prev [-] | | If a CS graduate can't figure out some simple gmail labels and filters then they should not be awarded that degree. Plain and simple. It's not rocket science. | | |
| ▲ | Poacher5 2 hours ago | parent [-] | | And there are no other students at any college other than CS students? I'm not sure why a biologist or a literature student would need to be au fait with Google's admittedly fairly unfriendly email management setup. | | |
| ▲ | denkmoon 38 minutes ago | parent [-] | | Digital literacy is important to every field. Email filters are not some arcane computer science concept, they are the modern equivalent of filing physical mail into the right folder/pidgeon hole/inbox/whatever. Biology is a great example because of just how important digital record management is to experimentation in the field. |
|
|
| |
| ▲ | fooker 4 hours ago | parent | prev | next [-] | | I have been using email for as long as email was a thing and I still managed to blackhole important emails with filters not too long ago. | |
| ▲ | setopt 2 hours ago | parent | prev | next [-] | | In my experience, it’s hard enough to make students check their school email in the first place. Let alone filter it. | |
| ▲ | gucci-on-fleek 4 hours ago | parent | prev | next [-] | | I'd hope/assume that any Computer Science students would be able to do this, but most Biology/Education/English/Art students probably couldn't. I mean, anyone smart enough to attend university could probably figure it out if they really wanted to, but there are hundreds of other useful things that they could learn too. There are only so many hours in the day, and given that most students don't get that many emails, I can hardly blame them for not wanting to prioritize learning how to filter emails. (I personally have over a hundred lines of Sieve filters, but I'm definitely not a typical student) | |
| ▲ | throawayonthe 2 hours ago | parent | prev | next [-] | | it's MS software, i think it's inanely difficult | |
| ▲ | shakna 4 hours ago | parent | prev | next [-] | | Most managers I've met, struggle with setting up email filters, and have to ask tech support to do it for them. These students will be qualified just fine. | |
| ▲ | emodendroket 3 hours ago | parent | prev | next [-] | | Most people who have office jobs don't know how to do this either | |
| ▲ | Scroll_Swe an hour ago | parent | prev | next [-] | | >Setting up custom email filters is beyond the capabilities of most students? Yes. And most of the general population. They can do it once they know it exists, most people just are not aware it is a thing at all. >What are they learning? Here, their "major" as you say in the US. Someone in econ, biology or even CS is not going to learn Outlook rules. Maybe IT or business will have a sentence on it. >Where will they be qualified to work? Any office job. Any job really. | |
| ▲ | mschuster91 3 hours ago | parent | prev [-] | | > What are they learning? Exactly what is in their field of study, nothing more. That's a huge part of the problems created by treating academia as a degree mill mandatory to get a job able to feed yourself instead of a place only for those truly interested in actually studying a subject. |
|
| |
| ▲ | e28eta 7 hours ago | parent | prev | next [-] | | Students having records of what their score was doesn't prove to the professor / university what score they received. "FWD: Exam 1 Results" is not especially auditable. | | |
| ▲ | lacunary 7 hours ago | parent | next [-] | | If only we had some way of signing messages | |
| ▲ | JumpCrisscross 6 hours ago | parent | prev | next [-] | | > Students having records of what their score was doesn't prove to the professor / university what score they received It's better than nothing. (And good training for the real world.) Also, most universities (and many schools now) issue academic e-mail addresses to students. In those cases, the email is definitive proof. | |
| ▲ | AmblingAvocado 6 hours ago | parent | prev | next [-] | | DKIM signature could be used to verify that Canvas' server sent the email with the given content | | | |
| ▲ | gruez 7 hours ago | parent | prev | next [-] | | As opposed to a screenshot of a website? Presumably the professor has a spreadsheet of all assignment grades that is submitted to the school? | | |
| ▲ | JumpCrisscross 6 hours ago | parent | next [-] | | > Presumably the professor has a spreadsheet of all assignment grades that is submitted to the school? This would undermine Canvas's lock-in. | | |
| ▲ | freeopinion 6 hours ago | parent | next [-] | | Canvas is built to automatically export its gradebook to an external system. It will do that automatically every day if you want it to. Teachers or others can manually export to the configured foreign system on demand. So if you grade something and want it to show up in the foreign gradebook without waiting for the daily export, you can just press the button to make it happen right away. | |
| ▲ | doctorpangloss 6 hours ago | parent | prev [-] | | i cannot believe how much benefit of the doubt people are giving canvas ed tech is the WORST performing VC sector the ONLY game in that town is vendor lock-in! are people joking? c'mon, canvas is a huge piece of shit. the SaaSpocalypse is coming for them - it seems it is simply that LLMs will be used to exploit it first, rather than universities writing an open alternative they share with each other for free. | | |
| ▲ | freeopinion 5 hours ago | parent | next [-] | | Canvas is AGPL licensed. Moodle is GPL. Universities or anyone else can already contribute to big name LMS. Canvas is used by Harvard, MIT, Stanford, Carnegie Mellon, CalTech, etc. If they each paid 10 FTE, they could set up a foundation that could govern the development of a top-tier LMS. Every tier-1 state institution could contribute 5 FTE. Even little JuCos could chip in an employee here and there. You'd pick up hundreds of capable employees at a fraction of what those schools currently pay to Instructure. | | |
| ▲ | freeopinion 5 hours ago | parent | next [-] | | How well has this worked for Open edX? | |
| ▲ | gizajob 2 hours ago | parent | prev [-] | | Why do they all pay for it then? Seems pretty universal in the UK too. Is it having the benefit of someone to blame when things go wrong? |
| |
| ▲ | freeopinion 5 hours ago | parent | prev | next [-] | | On paper your idea seems obvious. You take a bunch of institutions that actually teach students how to program and have them cooperate to build an open LMS that benefits them all. In reality, universities always spin off anything that looks like it could generate revenue. It is very telling that you can't even get your college transcript from your college. You have to go to (and pay) some third party to get it. Some universities even outsource their "classes" like elderhostel to cruise lines and travel companies. | |
| ▲ | gucci-on-fleek 5 hours ago | parent | prev [-] | | > rather than universities writing an open alternative they share with each other for free That already exists [0], and is actually reasonably popular. > the SaaSpocalypse is coming for them - it seems it is simply that LLMs will be used to exploit it first I doubt it, because enterprise sales has nothing to do with how good your product is, how expensive it is, how easy it is to administer, how secure it is, etc.; it only depends on how good you are at enterprise sales. I mean, my university is Oracle-based, and I'm pretty sure that you could get 3 random undergraduates to write something better, so I don't think that LLMs writing better/cheaper software will make any difference here. [0]: https://moodle.org/ |
|
| |
| ▲ | blahedo 7 hours ago | parent | prev [-] | | Nope! We're encouraged to keep all that exclusively in canvas. (As noted, I have my own spreadsheet. But I'm an outlier.) |
| |
| ▲ | gucci-on-fleek 5 hours ago | parent | prev | next [-] | | Presumably the system will be back up eventually, so there's not much benefit to lying here, since at best you'll raise your grade in a few classes for a couple months, while taking on a pretty big risk of getting caught. | |
| ▲ | pishpash 7 hours ago | parent | prev [-] | | You forget things can be signed, with the key owned by the school. It can be done. | | |
| ▲ | SlightlyLeftPad 7 hours ago | parent [-] | | Does signing really make this easily auditable from the professor’s perspective? | | |
| ▲ | DaSHacka 6 hours ago | parent [-] | | Exactly this, when was the last time a HN user had to interact with the prototypical 60-year-old set-in-their-ways professor? Extremely non-tech savvy, hates computers, and is gonna grumble "What the hell is a PGP? Better not be another one of those phone code things." as you try to pitch this highly-technological solution to a largely niche problem domain. | | |
| ▲ | jazzyjackson 4 hours ago | parent | next [-] | | I mean a cloud based learning management system also seems to be a very technological solution to the very old problem of checks notes grading quizzes? | |
| ▲ | Forgeties79 6 hours ago | parent | prev [-] | | They don’t even need to not be tech savvy. This stuff just registers as “hassle” to most people so they do the bare minimum or search for ways to not deal with it at all. It’s easy to “tut tut” at them but ultimately we need to accept reality: privacy, security, these things take extra effort that isn’t strictly necessary for people to go about their daily lives even though the stakes can be super high. It’s not a problem until it is, so they aren’t really barriers that require people to do the work. It’s like convincing someone who just simply doesn’t want to go out and buy/install a lock on their door to go do it, except it’s not even a one-time thing. Their door works fine. They can come and go as they please. It’s not until something happens that they maybe change their tune (and even then!) Hell just getting people to do secure passwords is a whole thing. |
|
|
|
| |
| ▲ | MarsIronPI 7 hours ago | parent | prev | next [-] | | Makes me glad I've always avoided doing my work on web platforms. When we used to have to make presentations in Google Slides I used to do them in Org-mode, then export to Sheets. I still have all those assignments sitting on my disk. Sure, there's versions of them on Google Drive, but I always make sure that the canonical version is the one on my disk. | |
| ▲ | moralestapia 6 hours ago | parent | prev [-] | | >It’s so simple to send an e-mail to the student ... What seems easy on hobby projects gets way more difficult at scale. Source: experience. |
|
|
| ▲ | setopt 2 hours ago | parent | prev | next [-] |
| Just to add one more data point, we also use Canvas at my university. The deadline for submitting who are eligible (i.e. passed compulsory assignments and labs) to take the exam was yesterday, and I couldn’t meet that deadline because Canvas went down. I usually do corrections offline so I have backups of my own evaluations, but these are courses with many teachers and many TAs, so Canvas is the way we sync our assessments. |
| |
| ▲ | p-e-w an hour ago | parent [-] | | I guess what surprises me the most is that it’s even legal for schools to outsource the core of what they do to some random tech company. Either way, they were under no obligation to adopt this garbage technology regardless of whether it’s available, so this is 110% on them. | | |
| ▲ | matsemann 12 minutes ago | parent [-] | | The alternative would be that each school develop their own platform for this, which also isn't very good use of their time and money? |
|
|
|
| ▲ | rupx 7 hours ago | parent | prev | next [-] |
| I work in the Education sector as IT. We don't know much else either. Everything we know has come from reddit threads / hackernews threads. There has been 0 official communication today indicating this was an attack, yet the login page was defaced by ShinyHunters. |
|
| ▲ | copperx 6 hours ago | parent | prev | next [-] |
| I don't understand what's the panic and doomerism about. Any competent IT team has backups and will be up and running as they go back to a state before the breach. This is HN. I'm disappointed that everyone is talking about losing grades and going back to pen and paper. I don't see how that could happen in 2026. And from the hacker's message itself, it's clear they want money in exchange for not releasing private info, not for the data itself. Do we live in a fear based culture? Why the panic? Even if everything was hosted on Instructure's infrastructure, it's all AWS. I'd be VERY surprised if there aren't multiple way to go back to a previous state. Most of the work and delay is to make sure they figure out where the breach occurred. |
| |
| ▲ | simonreiff 8 minutes ago | parent | next [-] | | I'm sure you're right. Across tens (hundreds?) of thousands of institutions worldwide, each one is exercising its well-written incident runbook that not only gets updated regularly but also is rehearsed constantly, just in case something like this happens. After all, what university IT department DOESN'T prepare obsessively for the moment when they need to restore all grades on all assignments for all courses from backup and fall over to the backup system for final exam administration in any required format specified by any professor, in the second week of May, on a non-negotiable schedule? There's absolutely nothing to worry about here. | |
| ▲ | yread 2 hours ago | parent | prev | next [-] | | Schools don't have competent IT teams. Here in the Netherlands a data center's power source (not even the machines) burnt down, data center is offline and University of Utrecht, one of the biggest universities here, is closed. Access passes don't work, work from home environment doesn't work, student information system is down, system for grading doesn't work. No failover for any of them (or maybe it was in the same DC?) https://nos.nl/artikel/2613485-storingen-in-hele-land-door-b... | |
| ▲ | mschuster91 3 hours ago | parent | prev [-] | | > Any competent IT team has backups Backups can be sabotaged (turned off or schedules manipulated) or compromised (say, by lateral movement). > Even if everything was hosted on Instructure's infrastructure, it's all AWS. AWS Backup isn't foolproof. Get your hands on administrator credentials as an attacker and suddenly the only thing between everything being gone for good and unrecoverable even for AWS is remembering to have put a permanent deletion protection on all resources in AWS Backup. |
|
|
| ▲ | dumbfounder 7 hours ago | parent | prev | next [-] |
| Maybe a hybrid approach. Scramble to create a final exam/project and give them the option to do pass/fail or a real grade, their choice. And then wish for the death of saas and a day where you can deploy your own software you can control and modify as you need. |
| |
| ▲ | Avicebron 7 hours ago | parent | next [-] | | What is the strategic response then? Assuming I'm a student and my grades are gone, and I want to graduate, shouldn't I pick pass/fail? Does a future employer look at pass/fail vs the grade? do they care? Are there even jobs that matter enough to care out there for them? This seems like, solving the problem but without actually seeing the broader goal or trajectory education is supposed to follow. | | |
| ▲ | hansvm 4 hours ago | parent [-] | | Most jobs I've had didn't care about a transcript in the slightest. It matters for future education and a small selection of jobs, and even them a few pass/fail courses won't cause any issues. It's not great if important, major-specific coursework is pass/fail, but usually you're not allowed to do that, so when it does come up you'll just have somebody ask what absurd situation (like this canvas thing) caused it. |
| |
| ▲ | flexagoon 2 hours ago | parent | prev [-] | | > day where you can deploy your own software you can control and modify as you need. Canvas is mostly FOSS https://github.com/instructure/canvas-lms |
|
|
| ▲ | 7 hours ago | parent | prev | next [-] |
| [deleted] |
|
| ▲ | SoftTalker 7 hours ago | parent | prev | next [-] |
| > they have airgapped backups and can be working as soon as they can spin up new servers ... and assuming they have a documented, tested, and trusted restore process. |
| |
| ▲ | yongjik 6 hours ago | parent | next [-] | | Reminds me of the incident last year when a South Korean government's server room caught fire, which contained the government equivalent of Google Drive, and the only backup was in the same room, and they all burnt down together. Some data was permanently lost, and then officers told reporters that multi-regional backup was not yet built because it was too hard at such a massive scale... of 858 TB. | | |
| ▲ | selcuka 3 hours ago | parent [-] | | > it was too hard at such a massive scale... of 858 TB There are probably many S3 buckets in existence that are bigger than that. Not saying that they should've used S3, but it's definitely possible configure multi-regional backup (and a government can afford it). | | |
| |
| ▲ | rayrey 7 hours ago | parent | prev [-] | | Ah yes the “recovery” part of the continuity plan. We tested that right? Right? |
|
|
| ▲ | camillomiller 2 hours ago | parent | prev | next [-] |
| To my European ears this just sounds like a disaster like this waiting to happen. God bless the annoying privacy OSS advocates and bureaucrats, I guess. |
|
| ▲ | jonstewart 7 hours ago | parent | prev | next [-] |
| Backups are definitely helpful in ransomwares, but before systems can be restored and brought back online, victim organizations still need to assess the scope of the breach, find the initial access vector, identify compromised accounts, and evict the threat actor. That can take time. |
| |
| ▲ | garciasn 7 hours ago | parent [-] | | I’m not certain, but it appears you’re giving Instructure a pass here, as if this is the first time they were hacked. But, it’s the second, by the same group. As a parent of kids who are impacted by this, I’m not super concerned about the data being held for ransom, but I sure as fuck am concerned about how much it’s going to cost the district to move to another provider. | | |
| ▲ | JumpCrisscross 6 hours ago | parent | next [-] | | > I sure as fuck am concerned about how much it’s going to cost the district to move to another provider Does Canvas have cybersecurity insurance? | |
| ▲ | MattSteelblade 6 hours ago | parent | prev [-] | | Not at all; standard IR procedure is scope -> containment -> eradication -> recovery. There is a fog right now; we don't know all the details. It seems to me that it's just as likely they weren't fully kicked out before or that the initial vulnerability wasn't remediated. You can't recover until the threat actor has been removed. |
|
|
|
| ▲ | vasco 7 hours ago | parent | prev | next [-] |
| > let classes that normally count for a grade just submit grades as pass-fail. Because what else can you do? Schedule a single exam and that's your grade for that subject? That's how it should work anyway, credits for work during semester (or worse attendance) are not needed to evaluate if someone learned the material, give them an exam and done. |
| |
| ▲ | goobatrooba 4 hours ago | parent | next [-] | | That's just bad outdated practice. It leads to cramming and less remembering than of the demand is for students to do work and show learning and effort throughout the year. | | |
| ▲ | matsemann 9 minutes ago | parent | next [-] | | Most courses I've taken have obligatory assignments that are pass/fail, and you have to pass a certain amount during the semester to take the final exam. But the grade is determined entirely of the final exam. Which to me seems the best way, you still have to learn throughout the year. Especially to avoid cheating this works nice. And as an aside, most people I know that did a year abroad in the US got 1-2 grades higher, as it was quite easy to just farm extra credits. | |
| ▲ | sayamqazi 2 hours ago | parent | prev [-] | | It has been my observation that most of the better students were the ones who would not put in work during the semester/year and cram at the end. |
| |
| ▲ | blahedo 7 hours ago | parent | prev | next [-] | | That's maybe something a school can do if exams are next week, or after. At my school, tomorrow is the last day of exams. Most of the students have left campus. There's no time or mechanism to schedule an(other) exam. | |
| ▲ | scubadude 3 hours ago | parent | prev | next [-] | | Then you're testing how good someone is at exams as much as anything | |
| ▲ | pishpash 7 hours ago | parent | prev [-] | | Exams have performance variance. Otherwise you're only getting a pass/fall signal in any case. | | |
| ▲ | vasco 7 hours ago | parent [-] | | Exams are the only fair way to evaluate if someone knows something (written or oral, in person). Take homes and attendance are just window dressing. |
|
|
|
| ▲ | ElenaDaibunny a minute ago | parent | prev | next [-] |
| [flagged] |
|
| ▲ | redsocksfan45 an hour ago | parent | prev | next [-] |
| [dead] |
|
| ▲ | aaron695 6 hours ago | parent | prev [-] |
| [dead] |