Arrowhead probably deserves more love for breaking the norm but I think it's overshadowed by people finding out for the first time the reason HDDs are so common in gaming setups is companies have been blindly shaving a few seconds off HDD load time off at the cost of 7x the disk space.

If it had been more well known this was the cause of game bloat before then this probably would have been better received. Still, Arrowhead deserves more credit both for testing and breaking the norm as well as making it a popular topic.

▲ abtinf 2 days ago | parent | next [-]

Part of what makes this outrageous is that the install size itself is probably a significant part of the reason to install the game on an HDD.

154GB vs 23GB can trivially make the difference of whether the game can be installed on a nice NVMe drive.

Is there a name for the solution to a problem (make size big to help when installed on HDD) in fact being the cause of the problem (game installed on HDD because big) in the first place?

▲ KronisLV a day ago | parent | next [-]

> 154GB vs 23GB can trivially make the difference of whether the game can be installed on a nice NVMe drive.

I think War Thunder did it the best:

  * Minimal client 23 GB
  * Full client 64 GB
  * Ultra HQ ground models 113 GB
  * Ultra HQ aircraft 92 GB
  * Full Ultra HQ 131 GB

For example, I will never need anything more than the full client, whereas if I want to play on a laptop, I won't really need more than the minimal client (limited textures and no interiors for planes).

The fact that this isn't commonplace in every engine and game out there is crazy. There's no reason why the same approach couldn't also work for DLCs and such. And there's no reason why this couldn't be made easy in every game engine out there (e.g. LOD level 0 goes to HQ content bundle, the lower ones go into the main package). Same for custom packages for like HDDs and such.

▲ consp 2 days ago | parent | prev | next [-]

Can any games these days be reliably ran on hdd's with max 200mb/s throughout (at best)? Or does everyone get a coffee and some cookies when a new zone loads? Even with this reduction that will take a while.

I thought all required ssd's now for "normal" gameplay.

▲

kbolino 2 days ago | parent [-]

Until you get to super-high-res textures and the like, the throughput isn't nearly as important as the latency.

At 200 MB/s the way hard drives usually measure it, you're able to read up to 390,625 512-byte blocks in 1 second, or to put it another way, a block that's immediately available under the head can be read in 2.56 microseconds. On the other hand, at 7200 RPM, it takes up to 8.33 milliseconds to wait for the platter to spin around and reach a random block on the same track. Even if these were the only constraints, sequentially arranging data you know you'll need to have available at the same time cuts latency by a factor of about 3000.

It's much harder to find precise information about the speed of the head arm, but it also usually takes several milliseconds to move from the innermost track to the outermost track or vice versa. In the worst case, this would double the random seek time, since the platter has to spin around again because the head wasn't in position yet. Also, since hard drives are so large nowadays, the file system allocators actually tend to avoid fragmentation upfront, leading to generally having few fragments for large files (YMMV).

So, the latency on a hard drive can be tolerable when optimized for.

▲

wtallis 2 days ago | parent [-]

> On the other hand, at 7200 RPM, it takes up to 138 microseconds to wait for the platter to spin around and reach a random block on the same track.

You did the math for 7200 rotations per second, not 7200 rotations per minute = 120 rotations per second.

In gaming terms, you get at most one or two disk reads per frame, which effectively means everything has to be carefully prefetched well in advance of being needed. Whereas on a decade-old SATA SSD you get at least dozens of random reads per frame.

	▲	kbolino 2 days ago \| parent [-]
		Fixed!

▲ jayd16 2 days ago | parent | prev [-]

"Self fulfilling prophecy" perhaps?

▲ nopurpose 2 days ago | parent | prev [-]

My immediate question is that if all of that was on-disk data duplication, why did it affected download size? Can't small download be expanded into optimal layout on the client side?

▲

braiamp 2 days ago | parent | next [-]

It didn't. They downloaded 43 GB instead of 152 GB, according to SteamDB: https://steamdb.info/app/553850/depots/ Now it is 20 GB => 21 GB. Steam is pretty good at deduplicating data in transit from their servers. They are not idiots that will let developers/publishers eat their downstream connection with duplicated data.

https://partner.steamgames.com/doc/sdk/uploading#AppStructur...

▲

myself248 2 days ago | parent [-]

Furthermore, this raises the possibility of a "de-debloater" that HDD users could run, which would duplicate the data into its loading-optimized form, if they decided they wanted to spend the space on it. (And a "de-de-debloater" to recover the space when they're not actively playing the game...)

The whole industry could benefit from this.

▲

nomel 2 days ago | parent [-]

> to recover the space when they're not actively playing the game

This would defeat the purpose. The goal of the duplication is to place the related data physically close, on the disk. Hard links, removing then replacing, etc, wouldn't preserve the physical spacing of the data, meaning the terrible slow read head has to physically sweep around more.

I think the sane approach would be to have a HDD/SDD switch for the file lookups, with all the references pointing to the same file, for SDD.

▲

myself248 a day ago | parent [-]

So you'd have to defrag after re-bloating, to make all the files contiguous again. That tool already exists, and the re-bloater could just call it.

▲

nomel a day ago | parent [-]

Sure, but defrag is a very slow process, especially if you're re-bloating (since it requires shifting things to make space), and definitely not something that could happen in the background, as the player is playing. Re-bloating definitely wouldn't be good for a quick "Ok, I'm ready to play!".

	▲	myself248 2 hours ago \| parent [-]
		I imagine it'd be equivalent to a download task, just one that doesn't consume bandwidth.

▲

ender341341 2 days ago | parent | prev | next [-]

depending on how the data duplication is actually done (like texture atlasing the actual bits can be very different after image compression) it can be much harder to do rote bit level deduplication. They could potentially ship the code to generate all of those locally, but then they have to deal with a lot of extra rights/contracts to do so (proprietary codecs/tooling is super, super common in gamedev), and

Also largely cause devs/publishers honestly just don't really think about it, they've been doing it as long as optical media has been prevalent (early/mid 90s) and for the last few years devs have actually been taking a look and realizing it doesn't make as much sense as it used to, especially if like in this case the majority of the time is spent on runtime generation of, or if they require a 2080 as minimum specs whats the point of optimizing for 1 low end component if most people running it are on high end systems.

Hitman recently (4 years ago) did a similar massive file shrink and mentioned many of the same things.

▲

ahartmetz 2 days ago | parent | prev [-]

Sure it can - it would need either special pre- and postprocessing or lrzip ("long range zip") to do it automatically. lrzip should be better known, it often finds significant redundancy in huge archives like VM images.