What thousands of conversations look like inside a language model

netruk44 an hour ago | parent | next [-]

I’ve been playing around with mapping steam games + their reviews based on Qwen embeddings. I revisit this project every year around this time to see what improvements I can make to make a “better” map than last year (According to my own preferences). [0/1]

One thing I learned this year is that naive text mapping (embed a corpus of reviews with a Qwen embedding model, use umap to plot, then compare where things fall) primarily compares embedded text length more than anything else. What a review says affects the position less than how long the review is.

This was a little surprising to me. For whatever reason, I didn’t think the embedding would so strongly describe the length of the embedded passage. So most of this year’s time was spent fixing that artifact.

[0]: 2025’s map: https://storage.danieltperry.me/share/steam-map/

[1]: 2026’s map (still a work in progress, both the ui and the review processing are incomplete): https://storage.danieltperry.me/share/steam-map-refresh/

▲

PaulHoule 2 minutes ago | parent | next [-]

Length-dependence has been a thing

https://en.wikipedia.org/wiki/Okapi_BM25

was one of very few useful results of the first decade of TREC research... And the main thing it addresses is balancing various factors which could cause a full text search engine to privilege either long or short documents.

▲

Jtarii an hour ago | parent | prev | next [-]

Interesting, a way to filter by number of reviews and review score would be nice.

	▲	netruk44 an hour ago \| parent [-]
		Yes, I agree. Those are coming. I only added the current filters and pins to the 2026 map yesterday.

▲

plagiarist 31 minutes ago | parent | prev [-]

This is an interesting idea, do you find it has more interesting properties than doing cosine similarly by reviewers? Have you found things adjacent to games you enjoyed that were also fun to play?

	▲	netruk44 11 minutes ago \| parent [-]
		Yes, I’ve actually found a few interesting games I had never heard of before! So the project has been a success for me already. But there’s still plenty of places where games are placed strangely on the map, and I like to investigate why that’s happening (it’s usually because Steam reviews are pretty noisy signal). The first version of this idea was actually based on the user entering arbitrary text or an existing Steam app id and looking at a list of games related by cosine similarity. It worked fine, but the results weren’t always closely related. It would be confusing to the user to get a top 10 list of related games that contained 2-4 obviously incorrect results. That approach was scrapped in favor of this map for two reasons: 1. This map doesn’t require I host an embedding model on the public internet. Static file hosting is my preference, which aligns better with a static map. 2. The list of games by cosine similarity can be recreated by looking at a game’s nearby points if the map is produced well enough that neighbors don’t move around too much. Plus with the map, you can see where it lies in relation to every other Steam game as well.

▲

harttrav 2 hours ago | parent | prev | next [-]

umap (https://umap-learn.readthedocs.io/en/latest/) or tsne (https://scikit-learn.org/stable/modules/generated/sklearn.ma...) might be an interesting alternative to pca for dimensionality reduction

▲

Scene_Cast2 2 hours ago | parent | prev | next [-]

The methodology feels either heavily undercooked to the point of being misleading, or heavily under explained. Why are we in 3D? At which point did dimensionality reduction hit? And if we're doing dimensionality reduction, does the "plane" really mean anything?

▲

vhold 2 hours ago | parent | next [-]

It's under explained. It's in 3D because of PCA. It mentions PCA all over the place, but never even says what it is, less alone briefly describe what it does.

https://en.wikipedia.org/wiki/Principal_component_analysis

There are a lot of more fancy ways to reduce dimensions and create visualizations like these that tend to preserve more of the higher dimensional structure.

Here's a semi recent paper discussing and comparing some of them, I didn't dig too deep to try to find a more recent or better one : https://pmc.ncbi.nlm.nih.gov/articles/PMC12453773/

▲

Scene_Cast2 an hour ago | parent [-]

I'm very familiar with PCA. There are so, so many ways you can apply it to this problem.

Assuming the simplest interpretation of what the analysis did, then it's saying "the variance of the biggest variance direction stays constant". Which in itself is an interesting thread to tug on, but the charts aren't how you'd present such a result.

	▲	thece0 an hour ago \| parent [-]
		Not familiar enough it seems. But PCA is outdated anyways

▲

oldjohnnylane 19 minutes ago | parent | prev [-]

There’s no meat here. It’s just someone throwing data into a plotting tool with no meaningful results or conclusions. I was expecting a meaningful analysis. Nope.

▲

rustyhancock 3 hours ago | parent | prev | next [-]

I wish there had been a third broad language style such as arabic although perhaps Qwen doesnt use arabic.

It made me think what if totally different knowledge bases could exist in a model where as my mental model was similar to OPs translation from pure knowledge source.

It also explains why rewriring prompts to another language sometimes lets you bypass restrictions.

▲

slickytail 3 hours ago | parent | prev | next [-]

I think the raw tokens embeddings shouldn't have been included in the PCA (past the first or second graph). Since they're disjoint from the conversation embeddings, you're essentially losing a dimension and compressing all the conversations into a plane. There may well have been much more obvious structure in the language embeddings that you'd be able to see.

▲

crib-award-cont 2 hours ago | parent | prev [-]

The language comparison angle is interesting; a broader set of languages would make this even more informative.