| ▲ | rdtsc 7 hours ago |
| > However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to SSL/TLS’s place in our business Scrutiny is inversely proportional to bit complexity of the private keys they used. Without a failure to apologize or expose how it happened: a human mistake, process mistake, etc. this looks pretty bad on them. > eMudhra’s global SSL/TLS infrastructure runs separately from its other operations. The company’s Global digital signature certificate (DSC), eSign, and other signing services, including those in India, its enterprise PKI Solutions (emCA, CertiNext platforms), emSigner eSignature Suite, and its identity and authentication solutions (SecurePass, emAS) are not affected and continue to operate as normal Anyone relying on them with anything related to security or trust is playing Russian roulette at this point. |
|
| ▲ | overtomanu 3 hours ago | parent | next [-] |
| It was "multiple human errors". But yeah no matter what, in any case or exceptional circumstances, this thing was not supposed to be screwed up. On Mozilla’s tracker, eMudhra said the keys were created on 30 April 2025 in a key generation ceremony, a recorded procedure in which keys are produced inside hardware security modules (HSMs), tamper-resistant devices built to hold keys. The keys were meant to be moved to data centres in the United States and Europe for customers needing data stored in those regions. To move them, the company added a step to ‘wrap’ each key, or encrypt it for transfer, and ‘unwrap’ it at the other end. eMudhra said a flaw in the software script created a second, weaker set of keys during this step, and because the two sets had similar names, staff used the weaker ones by mistake. It said human errors occurred at multiple steps. source: What's the eMudhra security scare that has rattled the internet? https://theprint.in/tech/whats-the-emudhra-security-scare-th... |
| |
| ▲ | wolvoleo 3 hours ago | parent [-] | | These "wrapping" keys were not "weaker". They were completely unfit for purpose. They have not explained why they used a 3-bit key for this purpose. If they go to the trouble of wrapping them, use real keys. Or don't do it. Also, it's a big "human error" to not notice this for a year and a half. | | |
| ▲ | silisili an hour ago | parent | next [-] | | I have to imagine that the 3 bit numbers were slots/indexes on the HSM, and rather than actually retrieving the key(or maybe it did), the slot number got used as the key. I can't otherwise imagine how they'd generate such low number keys... | |
| ▲ | patch_cable 2 hours ago | parent | prev | next [-] | | Having worked around these types of devices and their interfaces, I doubt they intended to create weak keys. It is more likely they had a script which placed a parameter in the wrong field, due to something like using positional arguments. Or they misread the documentation for some interface. Still incompetence. And I would love to see my theory confirmed or refuted. | | |
| ▲ | motionlessveloc 18 minutes ago | parent | next [-] | | The previous discussion advanced the theory that they fucked up the string handling. Eg in Python, output = key[0] would result in one-byte keys like "4" if you expect "key" to be an array, but it's actually a string. | |
| ▲ | oofbey an hour ago | parent | prev | next [-] | | I hear you in terms of how these kinds of mistakes typically happen. But why did the script ever produce two sets of keys? The initial root cause was somebody picked the weak key not the strong key. But what was the reason for ever producing a weak key in the first place? They say the script produced two outputs - what on earth for?? | | |
| ▲ | gopher_space 39 minutes ago | parent [-] | | Stepping back a little to look at the point and gravity of the exercise makes me wonder why they didn’t just fly it over. |
| |
| ▲ | lenerdenator 24 minutes ago | parent | prev [-] | | I mean, why have the ability to generate weak keys to begin with? Serious question, I've never done this sort of work. What sort of utility does that bring to the procedures and workflows one does in that line of work? If you didn't know better, you'd think this had been done on purpose by some three-letter agency. |
| |
| ▲ | rstuart4133 12 minutes ago | parent | prev | next [-] | | > These "wrapping" keys were not "weaker". They were completely unfit for purpose. Oh come on. They clearly didn't intend to use 4 as a private key or to wrap it. They had just purchased millions of dollars of HSM's to store them. It was something more subtle than that. Have a little imagination. The transfer procedure sounds bespoke - possibly just a shell script using openssl to encrypt things at one end, and decrypt them at the other. Let the man who has written 100 shell scripts without a bug in one cast the first stone. But not so subtle that good software engineering procedures wouldn't have picked it up. In fact today just an LLM review of the shell script probably would have caught it. The Internet Death Sentence was warranted - because poor software engineering procedures are unforgivable in a CA. But those procedures and policies aren't easy or cheap to come by. If they were just "don't chose 4 as a primary key" my grandkids could do it. | |
| ▲ | cogman10 2 hours ago | parent | prev | next [-] | | Yeah. I can't think of any development process where you wouldn't simply use a production grade key in development. In fact, that's exactly what we do in development with snake oil keys. Generating a key with "4" was probably done as a joke, but also is insane to have ever lived outside some devs computer. | |
| ▲ | rockdoe an hour ago | parent | prev | next [-] | | I don't think it's implied the wrapping keys were the weak ones. More like, the script that ran the wrapping still had the fast "debug" generation of a pair of keys to test the wrapper on enabled, for example. | |
| ▲ | telebub 2 hours ago | parent | prev [-] | | Unwrapped extraction is the kind of thing security by checkbox covers and process review might catch so it is never clear in a case like this that some party with the capability to influence the process didn't want keys intentionally weak to emulate unwrap.. Then someone else may have introduced the swap of keys that would never have been seen with keys that would (be forever) in a public scrutiny DB. |
|
|
|
| ▲ | sippingabonedry 4 hours ago | parent | prev | next [-] |
| Their complaint about social media reeks of "and we would have gotten away with it, if it wasn't for those meddling kids". [1] [1] https://www.youtube.com/watch?v=hXUqwuzcGeU |
| |
| ▲ | rdtsc 4 hours ago | parent | next [-] | | We deeply and sincerely apologize for getting caught and not being able to cover it up. | |
| ▲ | khriss 3 hours ago | parent | prev | next [-] | | This is pretty normal in India unfortunately. Crony capitalism is the dominant business engine in the country, so pretty much any significant business, esp in regulated sectors like govt mandated services(which is the bulk of eMudhra's operations) is impossible to set up without significant bribes being paid. As a consequence, these businesses are very well protected from normal market forces and are pretty much above the law. Had this incident not been an international one, they had a very good chance of getting away with it, and the people reporting this would have landed in jail. | | |
| ▲ | thraway3837 2 hours ago | parent | next [-] | | This is 100% accurate. State run media would have squashed this article if it was not for an international incident response. They have federal laws that prevent such disclosures because it affects the national image and could cause social unrest. | |
| ▲ | moralestapia 2 hours ago | parent | prev | next [-] | | Since you seem knowledgeable about this topic. Would you say the system forces people to be like that or people are the reason why the system is like that? Would you do business (i.e. trust) with someone that comes from that system? | | |
| ▲ | extraextra 4 minutes ago | parent | next [-] | | You're asking a chick and egg question, I believe it becomes a self-reinforcing loop In regards to doing business - someone that comes from a corrupt society with a culture of taking advantage of others should be approached more carefully than someone that grew up in the same town as you, of course This is a lesson learned the hard way, where I moved to a different country and treated everyone with openness and generosity (of time, resources, willingness to help), just to realize people from certain cultures mistake this behavior for weakness and try to take advantage as much as possible | |
| ▲ | qart 19 minutes ago | parent | prev [-] | | > Would you say the system forces people to be like that or people are the reason why the system is like that? It's the courts. Courts are meant to bring in an error correction mechanism. When courts fail, the country goes into open-loop control. No errors are corrected, and eventually, people stop even detecting errors. https://x.com/captgouda24/status/2110089855380369511 > Would you do business (i.e. trust) with someone that comes from that system? Vet rigorously and continuously! Start with small business, and grow it based on interpersonal trust. Since the system lacks accountability, you must ensure that yourself. |
| |
| ▲ | franktankbank 3 hours ago | parent | prev | next [-] | | Wild to me that our healthcare data lives there. | | |
| ▲ | lenerdenator 28 minutes ago | parent [-] | | Depends on the contract your healthcare providers/payers/etc. have with those who do their IT. Source: work on healthcare IT projects and can't let anyone outside of the US look at prod data no matter what due to the contract we have with our clients. |
| |
| ▲ | squidmaster 3 hours ago | parent | prev [-] | | [dead] |
| |
| ▲ | hnlmorg 4 hours ago | parent | prev [-] | | It does, but I wonder how much of that phrasing is cultural. When working with different businesses across the globe, I’ve noticed Indian correspondence to incidents tends to be more a show of strength even if it’s transparently obvious that there’s no merit to the content. Whereas western European correspondence is often more apologetic and empathetic. Even if it might sound insincere to those affected. But it’s all just words at the end of the day. It never means much unless the business is willing to back real change to mitigate similar issues in the future and resolve undisclosed existing problems. However there’s no way for us to know if this is happening or not. | | |
| ▲ | ryandrake 3 hours ago | parent [-] | | I don't know if it's really specific to one culture. Look at US companies after they go through a lengthy legal saga, it's obvious they screwed up, they're paying fines or having to adhere to court-imposed consequences. Their press releases: "We admit no wrongdoing. We are only settling to move past this. Our business is strong/secure/competent." It's like the first rule you learn when you go into corporate communications is to never admit you did anything wrong or project anything other than confidence and correctness. |
|
|
|
| ▲ | midtake 2 hours ago | parent | prev | next [-] |
| "out of proportion to SSL/TLS’s place in our business" is an insane comparison. They are saying they care so little about SSL/TLS to begin with that it's not fair. It would be like me practicing surgery 1 hour a week and saying "well surgery only has a very small place in my 80-hour work week" in response to flagrant failure of medical care. |
| |
| ▲ | cogman10 2 hours ago | parent [-] | | "We are bad at this job, but we don't do it often so don't think we are bad at other jobs". Yeah, the proportion if anything is too small. This is the sort of company that should not be doing anything related to security. |
|
|
| ▲ | pigggg 6 hours ago | parent | prev | next [-] |
| Various Indian regulations (particularly around commerce and banking) require you to use India companies for solutions like the above. It gets worse when companies are "steered" by a regulator/compliance person who has a bias for a specific solutions provider. |
| |
| ▲ | sippingabonedry 5 hours ago | parent | next [-] | | Then indirectly the security and trust of the entire Internet rests in the hands of a few government babus in India... that's scary. | | |
| ▲ | sandeepkd 4 hours ago | parent | next [-] | | For good or bad, thats by design. These is a significant portion of physical infrastructure associated with internet data flow. The only challenge governments face as of now is being able to segment the traffic to their wishes in its entirety. Companies already control all the traffic flow on their infrastructure using end-point security solutions, the same can be extended at ISP level, and then Country level | |
| ▲ | pigggg 4 hours ago | parent | prev [-] | | Their Internet registry people also weaponized AS0 (zero) as a method to take networks off air via BGP RPKI for the ranges they're responsible for. There's been a number of situations where multinational companies who operate in India who are forced to use IPs from the India RIR have been fat fingered by some staffer who makes changes then goes home for the day: meanwhile a multinational bank is down for 24 hours within the country because said agency doesn't believe in 24/7 support. Typically they do this for non pay but they've dropped the hammer multiple other times incorrectly (and good luck getting a RFO out of anyone there). |
| |
| ▲ | miohtama 2 hours ago | parent | prev | next [-] | | It’s called regulatory capture. It’s get worse when these companies work with regulators to create more regulation that they can sell more of their services. https://www.amazon.co.uk/Compliance-Industrial-Complex-Opera... | |
| ▲ | cylemons 5 hours ago | parent | prev | next [-] | | Regulation like this makes sense, but then the government should also srutinze these companies | | |
| ▲ | geodel 4 hours ago | parent [-] | | Well Indian regulator must have visited office canteen to see no unapproved food was served there. Only when fully satisfied they'd have given license to operate this security business. |
| |
| ▲ | debarshri 4 hours ago | parent | prev [-] | | "bias" |
|
|
| ▲ | gquere 5 hours ago | parent | prev | next [-] |
| Their OTP solution is bruteforceable as well: https://mastodon.catgirl.cloud/@49016/117475921584308861 |
| |
| ▲ | rdtsc 5 hours ago | parent | next [-] | | This becomes an inverse Russian roulette: only one of six chambers is not loaded. Waiting for a release like "... the level of social media amplification around this incident has brought scrutiny that is out of proportion to OTP 1FA place in our business. In view of this, after detailed deliberations our board has concluded that withdrawing from global OTP 1FA is the right decision for eMudhra and its shareholders" | | |
| ▲ | throw0101c 3 hours ago | parent [-] | | > This becomes an inverse Russian roulette: only one of six chambers is not loaded. At what point does it become Russian roulette with a Glock? | | |
| ▲ | boricj 3 hours ago | parent [-] | | One chamber is enough to play Russian roulette... once. However, the probability to get to play a second time is rather slim. |
|
| |
| ▲ | VorpalWay 2 hours ago | parent | prev | next [-] | | Link is broken, gives some randomly generated text when using Brave on Android. Their AI bot detection is not very good. | | |
| ▲ | rdtsc 2 hours ago | parent [-] | | > fear not, here's a complete bypass of their 1fa otp login system. they encrypt OTPs to a public key and issue those behind cloudflare turnstile. this key to be precise: |
| |
| ▲ | rauli_ 2 hours ago | parent | prev | next [-] | | That Web site thinks that my browser is an AI scraper for some reason. | |
| ▲ | anakaine 2 hours ago | parent | prev | next [-] | | It's not clear what you have linked there. | | |
| ▲ | rdtsc 2 hours ago | parent [-] | | Paste from mastodon: > fear not, here's a complete bypass of their 1fa otp login system. they encrypt OTPs to a public key and issue those behind cloudflare turnstile. this key to be precise: |
| |
| ▲ | thraway3837 2 hours ago | parent | prev [-] | | "However, the level of social media amplification around this incident has brought scrutiny that is out of proportion to OTP's place in our business" |
|
|
| ▲ | paustint 2 hours ago | parent | prev | next [-] |
| This is what I found regarding how it happened, but still not a full explanation - but there was some name confusion and obvious incompetence. >According to eMudhra, proper strong keys were created in the HSM at the key ceremony on 30-04-2025. The script had been changed to test wrapping and unwrapping keys, because the intermediates were to be moved to data centres in the US and Europe, and a bug in that change created a second set of weak keys. The keys had almost identical names, and the weak ones were used for the certificate requests. The same request was signed by eMudhra's own root and sent to Sectigo, which cross-signed it. The strong keys never made it into any certificate (Bugzilla). https://www.fairssl.se/en/articles/emudhra-ca-noegler-knaekk... |
| |
| ▲ | raffraffraff 2 hours ago | parent | next [-] | | Fantastic key ceremony. Great time had by all. Can't remember a damn thing, we were all so plastered. | |
| ▲ | CobrastanJorji 2 hours ago | parent | prev [-] | | I wonder. Say I wanted to prevent something stupid like this from happening in the future. What would be a sensible check? The higher level authority doesn't see the private key itself, right? Just the certificate request. And I wouldn't want someone to manually inspect the keys to see that it looks random-ish. Perhaps there could be a small check script added to the generation of certificate requests that says "hey, your key a.) is mostly zeroes, b.) appears to be ASCII text, or c.) seems otherwise extremely non-random, please use -f to continue anyway". | | |
|
|
| ▲ | cebert 5 hours ago | parent | prev | next [-] |
| Right? I don’t know why anybody would trust this company for anything. |
| |
| ▲ | fabioborellini 5 hours ago | parent [-] | | Someone higher up in the hierarchy telling you to trust this. You have no option to challenge it. |
|
|
| ▲ | saidnooneever 3 hours ago | parent | prev | next [-] |
| Russian Roulette is pretty accurate if not an understatement. If they expect people to still trust them on their other services they have obviously no clue on how Trust works, meaning they shouldn't be in the game in the frist place. As a trust provider you get one chance. Squander it and you're out. That is how it should be. |
|
| ▲ | kjs3 3 hours ago | parent | prev | next [-] |
| Anyone relying on them with anything related to security or trust is playing Russian roulette at this point. In any legal dispute involving them, Exhibit A will be "look at how they ran their SSL/TLS business". That won't go well. |
|
| ▲ | thraway3837 2 hours ago | parent | prev | next [-] |
| They got their image hurt, and pulling the culturally appropriate response here which is to never apologize and to never accept fault. Instead quit/resign and say that other stuff is just fine. It's really not fine, and you can bet that the same corporate philosophy that made the CA error will have serious security errors in their other products. |
|
| ▲ | MrBuddyCasino 3 hours ago | parent | prev [-] |
| > Without a failure to apologize Unsurprising if you are familiar with the indian honor code „izzat“. To quote an old kiwifarms thread: Hi. I didn't see this term mentioned in the thread, so I wanted to share this all with you. It's about Izzat. Some posts have mentioned 'saving face', which is Izzat, but I wanted to elaborate on it. Izzat has no direct translation into English. We only have terms that can broach the same concept such as 'honour' or 'reputation' or 'face'. Izzat is so much more than that. It's a zero-sum game of collective honour shared by whole groups of people, all of whom take it very, very seriously. A system like this isn't just foreign to Enlightenment values, but I'd argue it's antithetical to every sensible form of governance on the planet. It will destroy any system that assumes good faith. Izzat conflicts are not about who is right and who is wrong. It's about who wins and who loses. This means it's a zero-sum game where just about any action is justified (including murder) to restore the lost Izzat. Izzat is a limited social currency and the easiest way to get it is to take it from someone else. Winning is righteous in Izzat. Losing is unrighteous. This means that if someone plays the game of izzat well enough, they can get away with just about anything (murder, rape, scamming, cheating, stealing). The only morality in Izzat is the protection of your group's collective ego. The only appropriate response when your Izzat is attacked is the complete destruction of whoever insulted it. Two people get into an argument. They might escalate, but chances are it won't be that bad. i.e. 'Sorry I broke your garden gnome.' With two Indians in an argument, the stakes are always deadly thanks to Izzat. Neither of them can back down, nor can they admit fault. Admitting fault is seen as deliberately humiliating yourself. Not only that, but because Izzat is shared, you are shaming everyone who shares your Izzat. So, admitting fault or taking responsibility for a problem is a form of social suicide. This means even if the dispute was over something completely fucking stupid or trivial (like a broken garden gnome), it could spiral into a decades long honour feud.
Izzat is also the reason why police are reluctant to get involved in disputes. Because Izzat is a zero-sum game, all participants are involved in the game. If a policeman sides with one family over another in an Izzat dispute, this means that he's deliberately taking the side of that family and dishonouring the other. This marks him for retribution by the offending party. Without parties being able to be impartial, then centralized authority cannot effectively function. Let's say someone in India complains about a broken water pipe. Instead of the problem being addressed, the official responsible for the water pipe denies it's a problem and counterattacks him instead, because daring to question his efficacy in his role was challenging his Izzat. So the official destroys the person who brought the problem up. The water pipe never gets fixed. Izzat is also the reason why Indian managers are so infamous for hiring more Indians. It's because from the manager's perspective, he's using his position to gain an invisible social currency. Merit and actual qualifications come second to that idea. If he hired a westerner, he would not gain or even lose Izzat by doing so. He has a very strong cultural incentive not to be impartial. Meanwhile, if he can strong-arm dozens of Indians into a company, he is gaining huge quantities of Izzat and conspirators who owe an absolute debt to him. It's not uncommon to see Indians gloat about their success in the west. And yes, izzat is very much a system that enables short term success. But the fundamental reality is the prosperity that these Indians find so attractive in the first place wouldn't exist if the west practiced something similar to Izzat. Our systems can only exist on the assumption of good faith, and not a majority of people exploiting them for destructive short term gain. On top of that, if an Indian causes the systems and companies he comes into contact with to collapse, then he can just go back to India with his plunder. He has no stakes in the long-term prosperity, functionality or stability of these systems. The stakes are completely asymmetrical in the Indian's favour. You all know about that infamous video of that Indian scamming a food bank. Once again, winning is righteous. Losing is not. By employing 'clever' means (jugaad), he successfully extracted more resources for himself and his family. From the perspective of jugaad, a rule is not something to respect, but merely an obstacle to the Indian's own gain. Since he cheated the system and wasn't caught, he is seen as a righteous and dignified man in the Izzat framework. I read a story in India about a man offering to pay another man to use a public toilet instead of defecating openly. The latter left and came back with several friends and beat the former to death. The problem was not that he was defecating openly, but that he was criticised for doing so. And with Izzat, it's not an eye-for-an-eye. Izzat is often an eye-for-a-whole-head. Disproportionately and brutally annihilating your enemies is the correct move to make because you take back your lost Izzat and then some. This kind of vindictiveness would be rightly seen as horrifying and disgusting in most of the world. |
| |
| ▲ | throw93939e8 5 minutes ago | parent | next [-] | | West has something similar. Politicians never apologize, and avoiding Indian certificate authorities (and recuriters) is racisist!!! | |
| ▲ | anakaine 2 hours ago | parent | prev [-] | | Well reposted. |
|