Yesterday, I was asked to scrape data from a website. My friend used ChatGPT to scrape data but didn't succeded even spent 3h+. I looked website code and understand with my web knowledge and do some research with LLM. Then I described how to scrape data to LLM it took 30 minutes overall. The LLM cant create best way but you can create with using LLM. Everything is same, at the end of the day you need someone who can really think.

▲

jvm___ 4 days ago | parent | next [-]

LLM's can do anything, but the decision tree for what you can do in life is almost infinite. LLM's still need a coherent designer to make progress towards a goal.

	▲	demirbey05 4 days ago \| parent [-]
		LLMs can do small things well, but you must use small parts to form big picture.

▲

isatty 4 days ago | parent | prev | next [-]

Or you could’ve used xpath and bs4 and have been done in an hour or two and have more understandable code.

	▲	demirbey05 4 days ago \| parent [-]
		it is not that easy, there is lazy loading in the page that is triggered by scroll of specific sections. You need to find clever way, no way to scrape with bs4, so tough with even selenium.

▲

1718627440 4 days ago | parent | prev [-]

wget -m ?