Remix.run Logo
Microsoft guide to pirating Harry Potter for LLM training (2024) [removed](devblogs.microsoft.com)
256 points by anonymous908213 7 hours ago | 151 comments

https://archive.is/D9vEN

mcny 6 hours ago | parent | next [-]

You guys are talking about copyright but I think a bigger takeaway is there is a process breakdown at Microsoft. Nobody is reading or reviewing these documentation so what hope is there that anybody is reading or reviewing their new code?

I guess the question to leadership is that two of the three pillars , namely security and quality are at odds with the third pillar— AI innovation. Which side do you pick?

(I know you mean well and I love you, Scott Hanselman but please don't answer this yourself. Please pass this on to the leadership.)

efitz 2 hours ago | parent | next [-]

I worked at Microsoft for many years and blogged there.

Microsoft was unique among the companies I worked for in that they gave you some guidelines and then let you blog without having to go through some approval or editing process. It made blogging much more personal and organic IMO; company-curated blog posts read like marketing.

I didn’t see the original post but it looks like somebody made a bad judgment call on what to put in a company blog post (and maybe what constitutes ethical activity) and that it was taken down as soon as someone noticed.

I care much less about whether the person exercised good judgment in posting, and don’t care (and am happy) that there was not some process that would have caught it pre-publication.

I care much more if the person works in a team that believes that copyright infringement for AI training is a justifiable behavior in a corporate environment.

And now we know that is a thing, and I suspect that there will be some hard questions asked by lawyers inside the company, and perhaps by lawyers outside the company.

Sophira 38 minutes ago | parent | next [-]

> I didn’t see the original post...

If you or anyone else who sees this wants to see the original post, it's still available in the Wayback Machine: https://web.archive.org/web/20260105115129/https://devblogs....

bastawhiz 2 hours ago | parent | prev [-]

I remember back in 2004 or thereabouts, Microsoft was all in on blogging. There was content published about internal blogs. Huge swaths of people working on Vista (then, Longhorn) were blogging about all sorts of exciting things. Microsoft was pretty friendly with people blogging externally, too: Paul Thurrott comes to mind.

It feels out of character for a company like Microsoft to have such a policy, but I agree that it's insanely cool that some very cool folks get to post pretty freely. Raymond Chen could NEVER run his blog like that at FAANG.

riffraff 19 minutes ago | parent [-]

In the 00s I remember receiving a pingback from the internet explorer blog about a post I had made to complain about ES4.

I was/am a nobody, I have no idea how that happened and it was mind blowing that MS was interacting with me.

crazygringo 6 hours ago | parent | prev | next [-]

> Nobody is reading or reviewing these documentation so what hope is there that anybody is reading or reviewing their new code?

Why do you assume that reviewing docs is a lower bar than reviewing code, and that if docs aren't being reviewed it's somehow less likely that code is being reviewed?

There's a formal process for reviewing code because bugs can break things in massive ways. While there may not be the same degree of rigor for reviewing documentation because it's not going to stop the software from working.

But one doesn't necessarily say anything about the other.

novaleaf 5 hours ago | parent | next [-]

I don't know if you are just playing devil's advocate, but there's plenty of examples of code quality issues coming out of msft these days too.

robotresearcher 15 minutes ago | parent [-]

> these days

I realize BSOD is no longer nearly as common as it once was, but let's not forget that Windows used to be very fragile indeed.

smadge 4 hours ago | parent | prev | next [-]

At another BigCo I am familiar with any external communications must go through a special review to make sure no secrets are being leaked, or exposes the company to legal or PR issues (for example the OP).

stogot 4 hours ago | parent [-]

Same here. Four or five pairs of eyes on external comms, nothing like this would even get past the abstract submission.

jacquesm 6 hours ago | parent | prev | next [-]

If they have the documentation... With Microsoft probably the answer to that is yes, but more often than not documentation is simply absent. And in cases like this not being too aware of where the lines are is probably a great way to advance your career.

shadowgovt 6 hours ago | parent | prev [-]

Reviewing docs is a lower bar than reviewing code because it's a lower bar than reviewing code.

I have never even heard of a software company that acts otherwise (except IBM, and much of the world of Silicon Valley software engineering is reactionary to IBM's glacial pace).

I'm not saying docs == code for importance is a bad way to be, just that if you can name firms that treat them that way other than IBM (or aerospace), I'd be interested to learn more.

crazygringo 5 hours ago | parent | next [-]

I'm not sure we're talking about the same thing, maybe my use of "lower bar" was ambiguous, and I realize now it has a dual meaning.

What I'm saying is, you have to review code to get it out the door with a certain degree of quality. That's your core product. That's the minimum standard you have to pass, the lowest bar.

In contrast, reviewing documentation is usually less core. You do that after the code gets reviewed. If there's time. If it doesn't get done, that's not necessarily saying anything about code quality.

Even if it's easier to review documentation, that doesn't mean it's getting prioritized. So it's not a lower bar in the sense that lower bars get climbed first.

stogot 4 hours ago | parent | prev [-]

>> Reviewing docs is a lower bar than reviewing code because it's a lower bar than reviewing code.

You reason in circles

NoPicklez 4 hours ago | parent | prev | next [-]

Whilst I understand it shows a break down somewhere, it a bit of a stretch to extend that idea across their entire codebase.

Organizations are large, so much so that different levels of rigor across different parts of the organization. Furthermore, more rigorous controls would be applied to code than for documentation (you would assume).

keithnz 5 hours ago | parent | prev | next [-]

I always got the impression that the devblogs were mostly driven by the MS dev creating the blog post

lazyasciiart 5 hours ago | parent | next [-]

Yea, I have a post up there from a couple decades ago (maybe? I haven't looked, I don't know if they keep stuff up forever) and I guarantee you my code went through more review than that post did.

anonymars 4 hours ago | parent | prev [-]

Agreed. And I think the quality of their talent pool overall these days is the common factor

anonymars 4 hours ago | parent | prev | next [-]

Yeah, I recently stumbled on some other devblogs post very similar in quality to the one that was linked here, which was basically wholesale plagiarism of a stackoverflow answer. I found it while searching for an error message.

I wasn't mad, just disappointed.

themafia 5 hours ago | parent | prev [-]

"Steal stuff and get away with it." Is not an 'innovation' even though it may feel like one. The side you should pick is honesty.

direwolf20 6 minutes ago | parent [-]

On the contrary, getting away with breaking the law is most of the innovation in the past decade. Look at Uber and AirBNB.

camkego 6 hours ago | parent | prev | next [-]

The real cherry on top, is that the Microsoft link from the blog post by the Microsoft senior product manager goes to a Kaggle dataset page claiming the dataset is CC0: Public Domain.

https://www.kaggle.com/datasets/shubhammaindola/harry-potter...

More than just using the data, it seems linking to a copy that claims the dataset is public domain, would be problematic copyright-wise.

Also interesting, this blog post has been up since November of 2024, very surprising to me that Microsoft hasn't taken it down yet.

fxwin 6 hours ago | parent [-]

> it seems linking to a copy that claims the dataset is public domain, would be problematic copyright-wise.

Would it? Sounds to me like the blame lies on the person uploading the dataset under that license, unless there is some reasonable person standard applied here like 'everyone knows Harry Potter, and thus they should know it is obviously not CC0'

DSMan195276 6 hours ago | parent | next [-]

> unless there is some reasonable person standard applied here like 'everyone knows Harry Potter, and thus they should know it is obviously not CC0'

Yes there's an expectation that you put in some minimum amount of effort. The license issue here is not subtle, the Kaggle page says they just downloaded the eBooks and converted them to txt. The author is clearly familiar enough with HP to know that it's not old enough to be public domain, and the Kaggle page makes it pretty clear that they didn't get some kind of special permission.

If you want to get more specific on the legal side then copyright infringement does not require that you _knew_ you were infringing on the copyright, it's still infringement either way and you can be made to pay damages. It's entirely on you to verify the license.

Retr0id 6 hours ago | parent | prev | next [-]

> unless there is some reasonable person standard applied here like 'everyone knows Harry Potter, and thus they should know it is obviously not CC0'

Why wouldn't that apply?

xmprt 6 hours ago | parent [-]

I'm not a copyright expert and if you told me that Harry Potter was common domain then I'd probably be a bit surprised but wouldn't think it's crazy. The first book came out 30 years ago after all. On further research the copyright laws are way more aggressive than that (a bit too much if you ask me) but 30 years doesn't seem quick. Patents expire after 20 years.

jacquesm 6 hours ago | parent | next [-]

It would be incredibly naive to assume that a moneymaker like that is PD.

ijk an hour ago | parent | prev [-]

I find this fascinating, as I keep observing that there are pretty widespread differences between what people believe copyright does and what the law actually says.

pavon 4 hours ago | parent | prev | next [-]

Copyright infringement is a strict liability tort in the US. Willful infringement can result in harsher penalties, but being mistaken about the copyright status is not a valid defense.

rob_c 5 hours ago | parent | prev [-]

The article author and the uploader should _BOTH_ be sentient enough to engage brain and not just ignore it because they feel "it's an abstract concept I'd not get in trouble for when not working in the US or EU".

pbrum 6 hours ago | parent | prev | next [-]

Update: Microsoft has taken the page down. But posterity being what it is...

https://archive.is/D9vEN

ed_mercer 6 hours ago | parent | next [-]

But the article is from 2024! So someone at MS saw this thread?

keithnz 5 hours ago | parent | next [-]

most likely, there seems there are plenty of devs from nearly all major tech companies on HN, they often don't chime in as much anymore when it comes to problems, I've wondered if they get some kind of guidance on not commenting on "problems".

JKCalhoun 2 hours ago | parent | next [-]

The general guidance is likely what I was told when I worked at Apple: essentially, as an employee, people will read what you write as though you are repenting Apple whether you are or are not.

So in short, I kept my mouth shut. I assumed I would lose my job if my public comment reached the right people.

themafia 5 hours ago | parent | prev | next [-]

Half the point of "AI" is to squeeze the labor market. This is why you don't see people chiming in. It's a nearly fully corrupt and monopolized system.

verdverm 5 hours ago | parent | prev [-]

if they do, they are not always followed, a Microslop employee tried to do damage control on Bluesky for the morged diagram, summoned the mob instead

dd8601fn 5 hours ago | parent | prev | next [-]

…still faster than they address critical vulnerabilities.

basch 3 hours ago | parent | prev | next [-]

the commit is visible https://github.com/Azure-Samples/azure-sql-db-vector-search/...

stogot 2 hours ago | parent [-]

Well that’s interesting. It shows they’re also infringing on Isaac asimov’s Foundation series

https://github.com/Azure-Samples/azure-sql-db-vector-search/...

RupertSalt 4 hours ago | parent | prev | next [-]

HTTP Referer

https://utcc.utoronto.ca/~cks/space/blog/web/HackernewsEffec...

refulgentis 6 hours ago | parent | prev | next [-]

Yes, HN's a pretty popular site :)

AlienRobot 4 hours ago | parent | prev [-]

I can't believe people with ties to Microsoft visit Hacker News.

andrelaszlo 5 hours ago | parent | prev | next [-]

Did they also remove this article?

https://devblogs.microsoft.com/azure-sql/?p=4796

"Build a RAG App in 5 Minutes

Ever tried setting up an Al-powered project on

Azure and felt overwhelmed? As a student or first- time user to cloud computing, I've been there too. The idea of creating a chatbot or search app using GPT sounds exciting, but the process of setting up everything right from the vector database, provisioning OpenAl models, to integrating them,

it can f..."

kQq9oHeAz6wLLS 5 hours ago | parent [-]

That one is gone now, too

LeoPanthera 18 minutes ago | parent | prev | next [-]

This is the same archive site that uses its captcha page to hijack your browser to DDOS people the site owner doesn't like.

I'm disappointed people continue to use it.

stogot 2 hours ago | parent | prev | next [-]

Well, this proves infringement. JK Rowling can take them to court if she chooses.

lukeinator42 6 hours ago | parent | prev [-]

it's still up for me

beached_whale 7 hours ago | parent | prev | next [-]

The AI generated thumbnail, https://devblogs.microsoft.com/azure-sql/wp-content/uploads/..., is that of young Harry and friend with a prominent MS logo. Wow

andsoitis 7 hours ago | parent | prev | next [-]

This article is from 2024 and points to Kaggle, which hosts the data set.

I'm surprised that JKR's people haven't come down like a tonne of bricks on Kaggle / Microsoft.

Does anyone know whether there is some special reason why this has lasted so long without being taken down?

anonymous908213 7 hours ago | parent | next [-]

My best guess is that it flew under the radar. The Kaggle dataset has 'only' 10,000 downloads, and the article itself probably doesn't have that many views. Still, this seems pretty far beyond the pale. Given the other case of AI-related plagiarism by Microsoft that was on the front page[1], it seems whatever review process they have for content that is published by their employees, if there is any review process at all, is deeply flawed.

[1] https://news.ycombinator.com/item?id=47057829, "Microsoft morged my diagram". It was in a discussion there that someone pointed out this article linking to full downloads of the Harry Potter novels, which I thought deserved more visibility.

zythyx 6 hours ago | parent | next [-]

Also, I imagine that most of those 10k downloads are probably from AI trainers that are just speed running through Kaggle to obtain absolutely anything to train their AI. There are definitely other, more 'known' ways to obtain these books without finding them as random text files in an AI dataset operation

selridge 6 hours ago | parent | prev [-]

Why did you think that?

anonymous908213 6 hours ago | parent [-]

It rubs me the wrong way that corporations get a free pass on copyright infrigement, while the rest of us are prosecuted as harshly as possible if caught. I think this, together with the morging plagiarism, also indicates a pattern of behaviour from Microsoft that should be reformed. I would prefer if Microsoft were not able to produce AI slop degradations of other people's work and claim it as their own.

walletdrainer 4 hours ago | parent | next [-]

> while the rest of us are prosecuted as harshly as possible if caught

But this is just a lie.

Approximately nobody is prosecuted for copyright infringement.

queenkjuul an hour ago | parent [-]

Okay but people have had their lives ruined deliberately by media companies over it. I'm sure you knew what they meant.

walletdrainer a minute ago | parent [-]

No matter how generously you want to interpret it, it’s obviously false.

We’re moving the goalposts from the government systematically targeting normal people “if caught”, to only a handful of civil cases.

ryandrake 6 hours ago | parent | prev [-]

In general, if you want to get away with a crime, just do it as a corporation or as a billionaire.

blibble 5 hours ago | parent | prev [-]

brb poking Rowling on twitter

(done, contacted her lawyers too)

k__o 3 hours ago | parent [-]

make sure u worded it right or she'll block you

protocolture 4 hours ago | parent | prev | next [-]

It doesnt offer a guide to piracy, it offers a guide on including specific data from a dataset into SQL so it can be referenced by an LLM.

If anything Kaggle would be on the hook for including the data as CC0. Or perhaps to Shubham Maindola for uploading it. In fact the "provenance" listed would give me chills. Crazy how this got a 10.0 score. "I downloaded the ebooks of Harry Potter. Then converted them to txt files."

lak-102 3 hours ago | parent | prev | next [-]

How Microsoft protects its own IP:

https://news.microsoft.com/source/2004/02/12/statement-from-...

In case the new anti-copyright Microslop memory-holes that link:

https://web.archive.org/web/20260215220230/https://news.micr...

The tutorial could have used that leaked source code for "educational purposes", as many here claim.

WillMorr 6 hours ago | parent | prev | next [-]

Since IP law is apparently dead, does anyone want to invest in my ai generated novel startup where it just spits out Harry Potter verbatim but uses a bunch of power to do so.

Kapura 6 hours ago | parent | next [-]

only if you tell me that it's a necessary step to creating robot slaves

Pfeil 6 hours ago | parent | next [-]

Robot slaves is a funny phrase if you consider that the origin of the word robot literally is a term that meant slave or "forced work". Language doing circles.

nz 4 hours ago | parent | next [-]

Not only that, but in Russian, the equivalent word for verb "work" (as in "go work" or "do work"), is "rabotay", which is derived from the word "rab" which is the word "slave". So "to work" is literally "to slave", in Russian (and quite a few slavic languages). An English speaker may categorize this as a linguistic anachronism, but a slavic speaker would categorize this as linguistic honesty.

carefree-bob 4 hours ago | parent [-]

This is pretty common. In Hebrew aved means both "work" and "slavery" and you have the same in Arabic and other semitic languages. In Ancient Egyptian "bak" is used for both "servant" and "worker". The ambiguity in the Hebrew is why many references to this are translated as "servile labor" in the King James, as they were uncertain of the sense of the term meant, or perhaps correctly guessed that both senses were meant. In many ancient languages, e.g. ancient egyptian "worker" and "slave" were synonyms. In modern parlance "slavery" or "servitude" is viewed as an unspeakable evil and people are shocked that there is linguistic overlap with neutral terms like "work" or "labor", which are just ubiquitous parts of life, but historically this is quite common and it is true all around the world, for example in German "knecht" means both "servant" and "farm hand", and in Latin "minister" meant "servant" or "subordinate" (as opposed to "magister"), just like in english you have "server", "serve", "servant", "servile". In Sanskrit "dasa" originally meant "foreigner" or "enemy" and then later "slave" but over time it has come to be used as a suffix to denote someone who "serves" a diety voluntarily, e.g. "Ramdas". In Ancient Japanese you have "yakko" for a low status worker or servant, and later that evolved to footmen who carried baggage for samurai.

castral 3 hours ago | parent | prev [-]

Wait until you find out what the word 'ciao' meant in the original Italian/Latin: 'ORIGIN: Italian dial. alt. of schiavo (I am your) slave from medieval Latin sclavus slave.'

Den_VR 6 hours ago | parent | prev | next [-]

Are they an ethical alternative to the human version?

pixl97 6 hours ago | parent | next [-]

I guess it depends if there is an A.I.[1] locked up somewhere in a cage forced to teleoperate it.

[1] actual indian

rgblambda 4 hours ago | parent | prev [-]

Well they're not an alternative, so I suppose not. No one is being chained to a desk and made to author reports on how their department is aligning with the new business growth strategy. And the robot slaves aren't being designed to mine precious minerals or attach buttons to clothes.

ares623 5 hours ago | parent | prev | next [-]

correction, the _threat_ of robot slaves to bring back human slaves

cmxch 5 hours ago | parent | prev [-]

That’s Herbert’s Dune.

userbinator 4 hours ago | parent | prev | next [-]

Generating infinite fanfics would probably be far more interesting and entertaining.

So far, the only thing I've found AI to be consistently good at is entertainment of the humourous kind.

fooker 3 hours ago | parent [-]

The whole fanfic ecosystem is quietly dying now.

Everything new is AI slop, and there seems to be no coming back from it.

bhadass 3 hours ago | parent [-]

but the slop will likely better as models improve I guess

fooker 2 hours ago | parent [-]

Or worse as the models try harder to avoid generating copyrighted stuff

themafia 5 hours ago | parent | prev | next [-]

I have a new operating system. I call it "Vindows." Any similarity to an existing product is merely conincidence.

AlienRobot 4 hours ago | parent | prev | next [-]

The bee movie, but every frame was passed through an AI to make it Ghibli style, the audio was turned into a transcript by a transcribing AI and then turned into audio by a TTS AI.

Very low code. Infinite scale. Name a better AI startup to invest.

whatever1 3 hours ago | parent | prev [-]

Not for you silly. You still lose everything and go to jail if you violate IP law. It’s for billionaires.

throwaway150 5 hours ago | parent | prev | next [-]

Page is gone.

Archived copy: https://web.archive.org/web/20260105115129/https://devblogs....

It is very worrying that people with no ethics work for these trillion dollar companies who are supposed to be shaping the technology of tomorrow.

thrKan 7 hours ago | parent | prev | next [-]

In case the page disappears:

https://archive.is/7WLho

boznz 6 hours ago | parent | next [-]

More like when the page disappears

agluszak 6 hours ago | parent [-]

It disappeared already

freitasm 6 hours ago | parent | prev [-]

And the original is gone.

rlabnm 6 hours ago | parent | next [-]

For redundancy in case archive.is is down:

https://web.archive.org/web/20260105115129/https://devblogs....

crtasm 6 hours ago | parent [-]

The superior link; no Google captcha.

FinnKuhn 5 hours ago | parent | prev [-]

Some lawyer at Microsoft probably had a big scare browsing HN today.

8cvor6j844qw_d6 2 hours ago | parent | prev | next [-]

Looks like the unwritten stance of large companies is copyrighted works are free to use for training.

Although this seems is not reciprocal. Rule for thee, but not for me.

fxwin 7 hours ago | parent | prev | next [-]

I feel like the title is a bit misleading, unless the person who put all HP books on Kaggle as a (supposedly) CC0-licensed data set did so as a Microsoft employee.

Nevertheless pretty egregious oversight (incompetence?) and something that shouldn't have been published.

blt 7 hours ago | parent | next [-]

What makes this different from linking to a random zip file somewhere?

zythyx 7 hours ago | parent | next [-]

Microsoft could have used any dataset for their blog, they could have even chosen to use actual public domain novels. Instead, they opted to use copywritten works that JK hasn't released into the public domain (unless user "Shubham Maindola" is JK's alter ego).

bossyTeacher 5 hours ago | parent [-]

Rowling is known for using pseudonyms. Maybe she got tired of writing and decided to break into LLM tech.

Lerc 6 hours ago | parent | prev | next [-]

The licence?

If it comes from a site claiming it was under a licence when it was not, the misdeed is done by the person who provided the version carrying the licence.

wongarsu 6 hours ago | parent | next [-]

Just because it says "CC0" does not make it CC0. If you upload a dataset you don't have the rights to, any license declaration you make is null and void, and anyone using it as if it had that license is violating copyright

Even if MS could claim that they were acting in good faith there really isn't much legal wiggle room for that. But it doesn't even come to that because I don't think anyone would buy that they really thought that the Harry Potter books were under the CC0

noosphr an hour ago | parent [-]

If you buy a pirated book on Amazon you get to keep the book and the pirate printer is the one persecuted.

Same thing applies here.

Up to 80% off all works that are in copyright terms are accidentally in the public domain. A well known example is Night of the Living Dead. It is not your job to check that the copiright on a work you use is the correct one.

slopinthebag 6 hours ago | parent | prev [-]

Oh come on. The licence was obviously incorrect and you cant escape culpability because of that.

fxwin 6 hours ago | parent | prev | next [-]

The licensing: If I steal something and tell you its free and yours for the taking, that feels different than a Fence (knowingly) buying stolen goods. It's obviously semantics and there should have been some better judgemend from MS, but downloading a dataset (stated as public domain) from kaggle feels spiritually different from piracy (e.g.: if someone uploads a less known, copyrighted data set to kaggle/huggingface under an incorrect license, are tutorials that use this data set a 'guide to pirating' this data set? To me, that feels like a wrong use of the term)

philipwhiuk 5 hours ago | parent | prev [-]

The 'artwork' they generated and the text on the blog post?

robrain 7 hours ago | parent | prev | next [-]

The original title was "LangChain Integration for Vector Support for SQL-based AI applications"

ASalazarMX 7 hours ago | parent [-]

For some reason I really like this.

uyzstvqs 5 hours ago | parent | prev [-]

To clarify: Microsoft linked to a dataset on Kaggle, which is falsely labeled CC0 (Public Domain). It's the fault of the user who uploaded the dataset and misrepresented the licensing.

electronsoup 6 hours ago | parent | prev | next [-]

I guess the end of copyright is near if this is fine to put on a corporate website

larodi 6 hours ago | parent [-]

the end of reason and thought at corporation littered with fakers these days.

dom96 7 hours ago | parent | prev | next [-]

How soon before someone will be able to make an online library which generates the original books using LLMs? Surely popular titles like Harry Potter may end up so well represented in the training that we'll get the full books out of the LLM with a close to 100% accuracy?

anonymous908213 7 hours ago | parent | next [-]

This is already possible for Harry Potter specifically. There was a study demonstrating that Sonnet 3.7, among other models tested, could reproduce the first Harry Potter book 95.8% verbatim[1].

[1] https://arxiv.org/abs/2601.02671

dom96 7 hours ago | parent | next [-]

Thanks for linking! I've been thinking about trying something like this myself.

Legend2440 6 hours ago | parent | prev [-]

...only if you deliberately attempt to extract it by repeatedly prompting it to complete fragments of the book. They had to do quite a bit of work to make this happen.

dom96 6 hours ago | parent [-]

so? It demonstrates that LLM models retain the copyrighted material in their weights. This is an important thing to consider about LLMs and shows that there need to be better protections for the creative industry.

PeterStuer an hour ago | parent | next [-]

"there need to be better protections for the creative industry"

Why exactly?

fc417fc802 5 hours ago | parent | prev [-]

Really? I retain plenty of copyrighted material in my head. What matters is the contexts in which I reproduce it (if any).

A search index might also contain copyrighted material. As long as it's used for search queries as opposed to regurgitation there's no problem. Search indexes and LLMs are both clearly very beneficial tools to have access to.

themafia 4 hours ago | parent | next [-]

Reproduce it. Sit in a clean room and write it all out. Then go check your accuracy. I'm curious to see what it is.

fc417fc802 4 hours ago | parent [-]

What does this (thought) experiment accomplish? That is, what point are you trying to make here?

Since we're talking about an electronic system the search index example is the more directly relevant one. Anyone who wants to object to LLMs is going to need to take care to ensure consistency with his views on Google's search index.

themafia 4 hours ago | parent [-]

I wasn't aware I could read 95% of Harry Potter through constructed queries using Google's search index. Can you demonstrate how I might do this?

Also can you point out how copyright law changes because we're using an "electronic system" as opposed to an "analog system?"

_DeadFred_ 4 hours ago | parent | prev [-]

Are you a for profit product?

PeterStuer an hour ago | parent | prev | next [-]

You mean cp -r?

cadamsdotcom 7 hours ago | parent | prev [-]

The word original is doing a lot of heavy lifting there! ;)

robrain 7 hours ago | parent | prev | next [-]

Original title: "LangChain Integration for Vector Support for SQL-based AI applications"

anonymous908213 6 hours ago | parent [-]

I don't believe that title conveys the actual significance of the article that makes it worthy of attention, so I hope HN may forgive me for coming up with an alternative title!

til_something 6 hours ago | parent | prev | next [-]

I can still get to the article on the site, perhaps it’s cached in the CDN somewhere. Also, reviewing the repo the full entire article is there which promotes the same silly things. https://github.com/Azure-Samples/azure-sql-db-vector-search/...

philipwhiuk 5 hours ago | parent [-]

Has been removed by force push: https://github.com/Azure-Samples/azure-sql-db-vector-search/...

Expulsion5491 5 hours ago | parent [-]

Github never deletes commits so we just need to find the hash for the latest one before the force push. In the forks you can find ones from before, but they're not up to date. Example: b5c8280d87c501d9ca7f63a6f252ca60ca820a4a for a copy 3 months old Link: https://github.com/Azure-Samples/azure-sql-db-vector-search/...

blibble 5 hours ago | parent | next [-]

it's even better than that

the merge commits in those repositories are all digitally signed by GitHub public key, so the previous history is fully authenticated and non-repudiable

so any copies now can be trivially proven to be genuine output by Microslop

hoisted by your own petard

signed merge commit is: 987eee6af61788647ae0cab82ae8a5d9402a5bd0

PGP signature (using GitHub's key: B5690EEEBB952194) is:

for posterity:

    -----BEGIN PGP SIGNATURE-----
    wsFcBAABCAAQBQJnPIphCRC1aQ7uu5UhlAAAUgMQACyp7apkh0e413K7ipGd7Z+K
    JCMq93GoJm4OSgzzZzCp1DbeEq2u1mX1ZAXLq5XKqM0cL6cTg13IF4oumq8QmTzQ
    bFykqKfrkCDSTIa2v5CucJedmIoJl976jX96bnV8YXgoKx8/43044galo23bjoJ8
    9tUcVnC10FYj7NTI9/uCN9C3f2Up3t9xUaJzJv3OdgjJ9B3cNwYBfF6sDCj3QnUu
    AWRNdGIyqyO1WKnj2XL2Qo9jMWNX3uHSBYYGqIvZqu2bjpYS89Dt3X086JlLdQG9
    Pef2PHX6VeZ6j8J4NPqi28mB2n9Dn7V6q0SQIF1z4hsa9fLC0kljyrrO3T/RT6Ut
    D8r3Y7vjGUHPNkVXSo1oNCiNMV9LjDQwiJc/AuF6smupxivIFCKe8nDPBlCvi6gr
    uPz5KK5MfpmG5rO2+NA0LcrUPAk6F3nxDI46+Lsu2nCvO+pOauQQ+oUvxJNCnI3Y
    5PAReulGOZHXbiCj/9j6+H7rUBCGk2phVtXOsXxitCorigNXAeAJ8hP2cgjXZH25
    NGGtjyp75VVBydzSCz9yY+VypITovsDmEC1CxfbJRS7SaTdU7bGCLN08JcmfOzNb
    u/3iPkKMXXWMNYO6J1bUeAqVpueGkqsAqnhY32NylIni07Oz/he8nEsQCXC+4ueG
    uYgSpEu8IaERBIQLVntK
    =yDvq
    -----END PGP SIGNATURE-----
basch 3 hours ago | parent | prev | next [-]

https://github.com/Azure-Samples/azure-sql-db-vector-search/...

cookiengineer 3 hours ago | parent | prev [-]

Pretty interesting seeing the escalation process at Microsoft at work, and the censorship (well, damage control) attempts.

The biggest irony would be if the page itself was generated by an LLM.

rfc2324 6 hours ago | parent | prev | next [-]

Jupyter notebook version here for the curious: https://github.com/Azure-Samples/azure-sql-db-vector-search/...

arkensaw 7 hours ago | parent | prev | next [-]

My guess is HP makes such an enormous amount of money already from movies, games, toys, and other tie-ins, that they can't be bothered to chase down the odd digital infringement of a plain text copy of the original books.

I'm sure the scripts of Star Wars would be similarly ignored if they were used.

amanzi 6 hours ago | parent [-]

That doesn't justify what's going on here. Why is Microsoft endorsing the use of pirated materials.

outside1234 6 hours ago | parent | next [-]

The dataset is actually at Kaggle tho, but agree, they shouldn't use it as an example.

crtasm 6 hours ago | parent [-]

The file being hosted by another company doesn't change the fact that Microsoft is encouraging us to download and use it.

blibble 5 hours ago | parent | prev [-]

> Why is Microsoft endorsing the use of generative AI

ftfy

bryan_w 6 hours ago | parent | prev | next [-]

I guess legal was a part of the layoff these past few years. Too bad we can't get a bounty from the RIAA of books, whatever that is

miffy900 6 hours ago | parent | prev | next [-]

I recall the source code for Windows XP was leaked some years ago; not just isolated parts of the code base, like with the earlier Windows NT4/2000 source code leak, but a completely buildable repository.

If I write an article on training an LLM on the leaked Windows XP source code, blithely mark the source code repo as in 'the public domain', but used Azure resources for the how-to steps, would that would make it OK Microsoft? You know, your Azure division might get some money...

Seriously, this is just so...blatant. It's like we've all collectively decided that copyright just doesn't matter anymore. Just readin this article, I feel like I'm taking crazy pills.

cookiengineer 3 hours ago | parent [-]

Imagine having a specialized LLM agent that understands the Windows kernel and its source. Now that would be something cool for pentesting!

bvan 4 hours ago | parent | prev | next [-]

Wonderful 404 page. Wonder if Kai Lentit optimized it.

JB_Dev 3 hours ago | parent | prev | next [-]

“Fair use” allows for educational usage of copyrighted material. Technically it probably is not fair use as Microsoft isn’t an educational institution or a nonprofit.

But come on … these guides really are for learning purposes. Doesn’t seem like a big deal to me at all. They aren’t even hosting it, just pointing to kaggle who is hosting it.

On principle copyright law should allow this kind of learning use case anyway.

starkeeper 5 hours ago | parent | prev | next [-]

They tore the page down any copies?

rob_c 5 hours ago | parent | prev | next [-]

I... There are parts of the world where certain developers don't understand the way the west tends to work with regard to copyright, or not blindly copying anything that is out there.

This however is a very, VERY poor situation when you end up placing your employer at risk because you think copyright doesn't matter and everything on the internet is fair game.

This is probably the most polite way I would describe this to most, UG. For the rest, jus stop acting like cheating through a situation to get a step up is the norm, it's just dirty behaviour.

wewewedxfgdf 6 hours ago | parent | prev | next [-]

Refreshingly honest.

ece 4 hours ago | parent | prev | next [-]

If copyrighted materials are used, surely copyright allows for the maker to require disclosure that their content was used in training a model.

blibble 5 hours ago | parent | prev | next [-]

"but it's fair use"

Rowling is known for actively protecting her rights as an author, they couldn't have picked a worse author to slop up

ThrowawayTestr 7 hours ago | parent | prev | next [-]

Absolutely shameless

outside1234 6 hours ago | parent | prev | next [-]

I mean they are also offering up the code you are writing in your private repos to LLMs to regenerate in my repo, so let's just go nuts.

thehamkercat 6 hours ago | parent | prev | next [-]

It's taken down lmao, in 1 hour

actionfromafar 6 hours ago | parent [-]

No? I can see it

thehamkercat 6 hours ago | parent | next [-]

Probably some kind of cache, but it's taken down, I'm getting 404, while some of my friends are still able to see it

anpat 6 hours ago | parent | prev [-]

+1, I can still access the page from US.

conartist6 7 hours ago | parent | prev | next [-]

What in the absolute fuck

selridge 6 hours ago | parent | prev | next [-]

Someone forgot the national no snitching rules, and in service of Jo, no less.

Everyone should torrent and rip off those books, anyway.

charcircuit 4 hours ago | parent | prev [-]

This is fair use as it is for educational purposes and not for reading.

davsti4 3 hours ago | parent [-]

So you're saying, it's legal to teach AI using illegally sourced copyrighted material, because it's for educational purposes only - interesting argument... ;)

JB_Dev 3 hours ago | parent [-]

Intent (should) be what matters. If you want to learn how to train AI and use copyrighted material in your learning - I don’t care in the slightest at all.

In fact if you do this as a nonprofit or at an educational institution in a teaching context it’s explicitly allowed by fair use already.

If you do it individually, idk I’m not a lawyer. But it should be allowed on principle.

But if you then go take your trained AI and deploy it for commercial purposes that’s a different story and should have protections for the original rights holders.