| ▲ | Kiboneu 4 hours ago |
| Nah, you can snapshot every 15 minutes. The snapshot interval depends on the frequency of changes and their capacity, but it's up to them how to allocate these capacities... but it's definitely doable and there are real reasons for doing so. You can collapse deltas between snapshots after some time to make them last longer. I'd be surprised if they don't do that. As an aside, snapshotting would have prevented a good deal of horror stories shared by people who give AI access to the FS. Well, as long as you don't give it root....... |
|
| ▲ | john_strinlai 4 hours ago | parent | next [-] |
| >Nah, you can snapshot every 15 minutes. obviously you can. but, what is the actual snapshot frequency? like, what is the timestamp of the last known good snapshot? that is what matters. in any case, the comment you are replying to is a hypothetical, which correctly points out that even a day or two of lost edits is fine (not ideal, but fine). your reply doesnt engage with their comment at all. |
| |
| ▲ | Kiboneu 4 hours ago | parent [-] | | > the comment you are replying to is a hypothetical, which correctly points out that even a day or two of lost edits is fine (not ideal, but fine). your reply doesnt engage with their comment at all. I did engage, by pointing out that it wasn't relevant nor a realistic scenario for a competent sysadmin. (Did you read the OP?) That's a /you/ problem if you rely on infrequent backups, especially for a service with so much flux. > what is the actual snapshot frequency? like, what is the timestamp of the last known good snapshot? ? Why would I know what their internal operations are? | | |
| ▲ | john_strinlai 4 hours ago | parent [-] | | >I did engage, by pointing out that it wasn't relevant nor a realistic scenario for a competent sysadmin. >Why would I know what their internal operations are? i mean... you must, right? you know that once-a-day snapshots is not relevant to this specific incident. you know that their sysadmins are apparently competent. i just assumed you must have some sort of insider information to be so confident. | | |
| ▲ | Kiboneu 4 hours ago | parent [-] | | I think you are misreading my comments and made a bad assumption. The reason I'm confident is because this has been my bread and butter for a decade. | | |
| ▲ | john_strinlai 4 hours ago | parent [-] | | >The reason I'm confident is because this has been my bread and butter for a decade. my decade of dealing with incompetent sysadmins and broken backups (if they even exist) has given me the opposite of confidence. but im glad you have had a different experience | | |
| ▲ | Kiboneu 3 hours ago | parent [-] | | > my decade of dealing with incompetent sysadmins and broken backups (if they even exist) has given me the opposite of confidence. Oh, I agree that the average bar is low. That's part of the reason I do it all myself. The heuristic with wikimedia is that they've been running a PHP service that accepts and stores (anonymous) input for 25 years. The longetivity with the risk exposure that they have are indicators that they know what they are doing, and I'm sure they've learned from recovering all sorts of failures over the years. Look at how quickly it was brought back up in this instance! So, yeah. I don't think initial hypothetical counterpoint holds water, and that's what I have been pointing out. | | |
| ▲ | jibal 2 hours ago | parent [-] | | Kudos for very polite responses to trolling. | | |
| ▲ | Kiboneu 2 hours ago | parent | next [-] | | I have good faith, though I should get off hn now... :P I still don't need to assume what the intent is. Troll or no troll, it works. My comments might inspire someone else to try a CoW fs. I'm also really impressed with wikimedia's technical team. | |
| ▲ | john_strinlai 2 hours ago | parent | prev [-] | | no one is trolling in this comment chain. i found kibone's reply to a hypothetical musing as if it was some counterpoint in a debate instead of a simple expansion on their comment to be off putting. we had some comments back and forth and we both came out of it just fine. weird of you to add on this little insult to an otherwise pretty normal exchange. | | |
| ▲ | Kiboneu an hour ago | parent [-] | | FWIW I did not assume that you were trolling, and yes we did come out fine. |
|
|
|
|
|
|
|
|
|
| ▲ | sobjornstad 4 hours ago | parent | prev | next [-] |
| Nowadays I refuse to do any serious work that isn't in source control anywhere besides my NAS that takes copy-on-write snapshots every 15 minutes. It has saved my butt more times than I can count. |
| |
| ▲ | Kiboneu 4 hours ago | parent [-] | | Yeah same here. Earlier I had a sync error that corrupted my .git, somehow. no problem; I go back 15 minutes and copy the working version. Feels good to pat oneself in the back. Mine is sore, though. My E&O/cyber insurance likes me. |
|
|
| ▲ | gchamonlive 4 hours ago | parent | prev [-] |
| The problem isn't the granularity of the backup but since the worm silently nukes pages, it's virtually impossible to reconcile the state before the attack and the current state, so you have to just forfeit any changes made since then and ask the contributors to do the leg work of reapplying the correct changes |
| |
| ▲ | Kiboneu 4 hours ago | parent [-] | | Why would nuked pages matter? Snapshots capture everything and are not part of wikimedia software. | | |
| ▲ | gchamonlive 3 hours ago | parent [-] | | The nuke might be legitimate? | | |
| ▲ | wizzwizz4 2 hours ago | parent [-] | | That's not a lot of state lost. Destructive operations are easier to replay than constructive ones. | | |
| ▲ | gchamonlive an hour ago | parent [-] | | Is Wikimedia overreacting then? | | |
| ▲ | wizzwizz4 an hour ago | parent [-] | | No: from what I can tell, they're being conservative, which is appropriate here. Once you've pushed the "stop bad things happening" button, there's no need to rush. |
|
|
|
|
|
