| ▲ | schleck8 3 hours ago |
| The HTTP traffic from Yandex is down to ~ 0 % after the third strike, after the first two strikes already degraded it. I wonder how this looks like at the network level because their search page still serves, but presumably their crawler is inactive now Edit: Here is some more info, they do have availability issues as is to be expected > Because three of its primary facilities were knocked offline within four days, Yandex’s failover mechanisms broke down[1][8]. > Outages have cascaded across Yandex Go (taxis), Yandex Pay, food delivery, smart home platforms, internal logistics, and third-party corporate services hosted on Yandex Cloud[2][7]. Problems have spilled into neighboring countries using these systems (Belarus, Kazakhstan, Armenia, Serbia)[2][7]. Edit 2: Here is the Yandex Cloud status page, their GCP/AWS equivalent. The entire compute core is offline, and even worse, Cloud Backup https://status.yandex.cloud/ru/dashboard > ru Compute Cloud, ru Kubernetes, ru PostgreSQL, ru ClickHouse, ru MySQL, ru YDB, ru Kafka, ru Application Load Balancer Interestingly, Yandex tells its users on the status page to switch to competitors Selectel, K2Cloud, and VK Cloud. Which will probably be hit next |
|
| ▲ | Bluestein 3 hours ago | parent | next [-] |
| This will be a case study in catastrophic events for HA and Ops folks.- |
| |
| ▲ | walrus01 2 hours ago | parent | next [-] | | I don't know how useful it will be as a case study since an HA/Ops person in (let's say, the US, Canada, AU, western european countries) has access to do things like buy hypervisor capacity or colocation in just about any location. I could go buy colo in Brussels or Panama City or San Diego or Edmonton or Melbourne tomorrow by contacting the right people. The Russians are much more restricted from doing anything geographically outside of their own borders (for very well justified sanctions reasons), so their limitations are not the same as an HA person in almost any other country. Maybe some Iranian HA/Ops people will be comparing notes with them. | | |
| ▲ | ivan_gammel 2 hours ago | parent | next [-] | | Sometimes there exist data or processing residence requirements that bind you to a specific region (e.g. EU-only). Also, „all AZs in this region no longer exist“ is not on everyone‘s bingo card, this is a good reminder why multi-AZ deployment in the same region may be not enough. For data center guys this is definitely a case study, because drone attacks are probably on the threat list for everyone now, for various reasons, including domestic terrorism and hybrid warfare. If you want to defend your investment, time to think what covers you from the air. | | |
| ▲ | walrus01 2 hours ago | parent [-] | | It's an open question at what point we'll start seeing small fiesty startups that want to sell you a privately owned air defense system for your datacenter roof. The tech to do it is probably far ahead of the regulatory regime to own and operate autonomous small radar+SAM batteries. | | |
| ▲ | ivan_gammel an hour ago | parent [-] | | It will require some involvement of state, but it is plausible, that critical infrastructure will be required to have this covered and certain big, licensed private contractors may be able to operate air defense systems on such sites (do you smell the money and lobbying effort?). Also this may influence the design of such data centers. A drone that a paramilitary group can build is not very sophisticated and won’t carry big payloads. Put an outer concrete shell or some better passive defense, and maybe that will be enough. | | |
| ▲ | walrus01 43 minutes ago | parent [-] | | As a result of some of my work only slightly related to telecom/internet infrastructure, I happened to get on the marketing lists for about six different Chinese companies that will gladly sell you a full size Shahed-136 clone complete with 2 stroke petrol motor. I wouldn't say that small paramilitaries being only able to fly a 5-10kg max size quadcopter with a munition on it into a datacenter is the limit of possible future capabilities. These things are cheaply made in fiberglass molds and look to be dead simple. They're already showing demo videos of them flying around with very low cost ardupilot capable flight controllers. They even advertise the airframe and motor packages on Instagram.... | | |
|
|
| |
| ▲ | themgt an hour ago | parent | prev [-] | | The DR plan for pros, "I could go buy colo tomorrow by contacting the right people" | | |
| ▲ | walrus01 an hour ago | parent | next [-] | | It was clearly meant as an example of the ease of a non-russian company buying geographically-distributed services in the global marketplace, not "I could go buy this with a few docusign signatures and SWIFT wire transfers tomorrow". I am not in the market for colo in Panama City or Melbourne. I don't think it's a bad thing that the russians are hampered by many limitations but whatever lessons their HA people are learning these days are probably very specific to a sanctioned Russia. | |
| ▲ | SOLAR_FIELDS an hour ago | parent | prev [-] | | Unironically yes, if you're just a typical B2B SaaS who doesn't have "datacenter getting hit by a literal missile" in your threat model, this coupled with offsite backups is probably the logical DR plan for that scenario. |
|
| |
| ▲ | cowlevel an hour ago | parent | prev | next [-] | | I think the complete destruction of the AWS Bahrain region got there first. | |
| ▲ | AlienRobot 2 hours ago | parent | prev | next [-] | | I don't think there is much you can do about being targeted by a military, simply because of how unique the attack vector is. Hacking is smart but happens through software, so if you have hardware (i.e. physical) barriers, that stops the hacking. Natural disasters happen physically but they are dumb, so if you have software redundancy (i.e. by copying the data to multiple locations), that stops them from destroying everything. A military is both physical AND smart. They can target multiple locations at once and destroy whole buildings. | | |
| ▲ | jacquesm 2 hours ago | parent | next [-] | | > I don't think there is much you can do about being targeted by a military, simply because of how unique the attack vector is. You could choose not to invade other countries. | | |
| ▲ | loeg 2 hours ago | parent | next [-] | | This (as with defense) is kind of outside Google or Yandex's purview. | | |
| ▲ | phatfish 36 minutes ago | parent | next [-] | | Yandex is a part of Russia's military. Just the same as Google, MS, Amazon etc. provide services for the US military. Google et al. are probably in the critical path of defence for most European countries as well, insane though that is. | | |
| ▲ | ivan_gammel 18 minutes ago | parent [-] | | Not defending Yandex here, but I think their military involvement might be as accidental as of an American farmer whose beef landed in the cantina of aircraft carrier. Government and military procurement in Russia is its own universe, with strong players having specific IT expertise and capable of building DCs and tech. Assuming that everywhere in the world it’s the same as in America is wrong. Consumer tech companies aren’t operating this way usually, it is the scale of American Big Tech that allows them to compete with traditional contractors. |
| |
| ▲ | jacquesm 2 hours ago | parent | prev [-] | | Said every company serving fascists ever. No, you can choose to collaborate or you can shut it down. Collaborators don't get to point back in time and say 'oh we had no choice', even if the choice was a painful one. Sorry it does not work that way. |
| |
| ▲ | AlienRobot 2 hours ago | parent | prev | next [-] | | I guess, but that doesn't sound like something in the control of a single individual company. | | | |
| ▲ | whatdouthink 2 hours ago | parent | prev [-] | | But apparently this does not apply to you. | | |
| ▲ | jacquesm 2 hours ago | parent [-] | | At least one russian shill that isn't affected by the Yandex strikes. Let's see how the rest of them fares, HN has its fair share of these characters. |
|
| |
| ▲ | simondotau 2 hours ago | parent | prev [-] | | You can learn to quickly diversify availability (or at least data safety) beyond the region of conflict when a kinetic war breaks out. The moment “kinetic sanctions” became the strategy, the risk to data centres should have been self-evident. And if it wasn’t immediately obvious, it should have been blindingly so when targeting expanded from oil refineries to Wildberries warehouses. | | |
| ▲ | ivan_gammel 2 hours ago | parent [-] | | I don‘t think you can build a new data center in such a short time frame that passed since attacks on WB, definitely not when you can source hardware only from the black market. | | |
| ▲ | simondotau an hour ago | parent [-] | | I’m not describing what Yandex should do, but rather what their customers should do. Whether that means diversifying to eastern Russia, or China, or Belarus, I don’t know what options exist. |
|
|
| |
| ▲ | 3 hours ago | parent | prev | next [-] | | [deleted] | |
| ▲ | Hamuko 2 hours ago | parent | prev [-] | | I feel like this isn't even a particularly new thing. I worked at a medium-sized SaaS company and we did disaster recovery drills for moving our stack from eu-west-1 to eu-central-1 in case something catastrophic happened in Ireland. If you want HA, you kinda have to assume that any single country in the world can have some kind of a crippling disaster. Unfortunately for Yandex's clients, Yandex only has regions in two countries, and not that many regions either. |
|
|
| ▲ | alephnerd an hour ago | parent | prev | next [-] |
| > The entire compute core is offline, and even worse, Cloud Backup Their Kazakh regions are available according to the dashboard, so it isn't as catastrophic. Their managed DBs and data processing services are heavily affected though with no Kazakh redundancies from the looks of it. |
|
| ▲ | bamboozled 3 hours ago | parent | prev [-] |
| I just searched "blyat" and it worked, so? |
| |
| ▲ | eptcyka 2 hours ago | parent [-] | | Given that blyad is just punctuation at this point, I am not surprised. | | |
| ▲ | throw-the-towel 2 hours ago | parent | next [-] | | The username checks out! ("Ept" and "suka" are two other swearwords, that are also used much like punctuation.) | |
| ▲ | walrus01 2 hours ago | parent | prev [-] | | I predict a great many more videos coming out of Russia of Firepoint UAVs flying into refineries and such, punctuated by background audio commentary that can be summed up as "cyka blyat" |
|
|
