Remix.run Logo
Sytten 4 hours ago

I used to believe that it was not necessary until I started building my own startup. If you dont have analytics you are flying blind. You don't know what your users actually care about and how to optimize a successful user journey. The difference between what people tell you when asked directly and how they actually use your software is actually shocking.

embedding-shape 4 hours ago | parent | next [-]

You're only flying blind if you make decisions not looking and thinking. Analytics isn't the only way to figure out "what your users actually care about", you can also try the old school way, commonly referred to as "Talking with people", then after taking notes, you think about it, maybe discuss with others. Don't take what people say at face value, but think about it together with your knowledge and experience, and you'll make even better product decisions than the people who are only making "data driven decisions" all the time.

johnfn 4 hours ago | parent | next [-]

Sure, you can spend the weeks to months of expensive and time consuming work it takes to get a fuzzy, half accurate and biased picture of what your users workflows look like through user interviews and surveys. Or you can look at the analytics, which tell you everything you need to know immediately, always up to date, with perfect precision.

Sometimes HN drives me crazy. From this thread you’d think telemetry is screen recording your every move and facial expression and sending it to the government. I’ve worked at places that had telemetry and it’s more along the granularity of “how many people clicked the secondary button on the third tab?” This is a far cry from “spying on users”.

graphememes 32 minutes ago | parent | next [-]

You're never going to win this argument, most of the people who post here have never actually shipped a product themselves and only work on isolated features and others have to handle / manage all of this for them so they have no real understanding of what it takes to do it

the other crowd that pretends otherwise are larping or only have some generic open source project that only a handful of people use or they only update it every 6 years

embedding-shape 3 minutes ago | parent [-]

> You're never going to win this argument

Probably because there is no "truth" here, only subjective opinion, there is no "winning", only "learning" and "sharing".

I could ramble the same about how "people relying on data never shipped an enjoyable thing to people who ended up loving, only care about shipping as fast as possible" and yadda yadda, or I can actually make my points for why I believe what I believe. I do know what I prefer to read, so that's what I try to contribute back.

embedding-shape 4 hours ago | parent | prev | next [-]

> Sure, you can spend the weeks to months of expensive and time consuming work it takes to get a fuzzy, half accurate and biased picture of what your users workflows look like through user interviews and surveys. Or you can look at the analytics, which tell you everything you need to know immediately, always up to date, with perfect precision.

Yes, admittedly, the first time you do these things, they're difficult, hard and you have lots to learn. But as you do this more often, build up a knowledge base and learn about your users, you'll gain knowledge and experience you can reuse, and it'll no longer take you weeks or months of investigations to answer "Where should this button go?", you'll base it on what you already know.

johnfn 2 hours ago | parent | next [-]

You seem to be interpreting my position as saying that one should only use telemetry to make decisions. Of course, no one reasonable would hold that position! What I’m saying is that only relying on user interviews without supplementing them with analytics would be knowingly introducing a blind spot into how you understand user behavior.

embedding-shape an hour ago | parent [-]

Yes, probably because someone else said "If you dont have analytics you are flying blind" which I initially replied to, then when you replied to my reply, I took that as agreeing with parent, which isn't necessarily true.

> What I’m saying is that only relying on user interviews without supplementing them

I also took your "spend the weeks to months of expensive and time consuming work [...] Or you can look at the analytics" as a "either this or that proposition", where if we're making that choice, I'd go with qualitative data rather than quantitative, regardless of time taken. But probably it comes down to what tradeoffs we're willing to accept.

3 hours ago | parent | prev | next [-]
[deleted]
hombre_fatal 3 hours ago | parent | prev | next [-]

Asking users isn't a substitute for usage data.

Usage data is the ground truth.

Soliciting user feedback is invasive, and it's only possible for some questions.

The HN response to this is "too bad" but it's a thought-terminating response.

AlotOfReading 3 hours ago | parent | next [-]

It goes the other way as well. Usage data isn't equivalent to asking users either. A solid percentage of bad decisions in tech can be traced to someone, somewhere forgetting that distinction and trusting usage data that says it's it's okay to remove <very important feature> because it's infrequently used.

hombre_fatal 44 minutes ago | parent | next [-]

Yeah, it's not a good discussion without concrete examples.

One: Building a good UX involves guesswork and experiments. You don't know what will be best for most users until you try something. You will often be wrong, and you rarely find the global maximum on the first try.

This applies to major features but also the most trivial UI details like whether users understand that this label can be clicked or that this button exists.

Two: Like all software, you're in a constant battle to avoid encumbering the system with things you don't actually need, like leaving around UI components that people don't use. Yet you don't want to become so terse with the UI that people find it confusing.

Three: I ran a popular cryptocurrency-related service where people constantly complained about there being no 2FA. I built it and polished a UX flow to both hint at the feature and make it easy to set up. A few months later I saw that only a few people enabled it.

Was it broken? No. It just turns out that people didn't really want to use 2FA.

The point being that you can be super wrong about usage patterns even after talking to users.

Finally: It's easy to think about companies we don't like and telemetry that's too snitchy. I don't want Microslop phoning home each app I open.

But if we only focus on the worst cases, we miss out on the more reasonable cases where thoughtful developers collect minimal data in an earnest effort to make the UX better for everyone.

ragall 23 minutes ago | parent [-]

> You don't know what will be best for most users until you try something.

That's because you don't understand your users. If you did, you wouldn't need to spy on them.

> you rarely find the global maximum on the first try

One never finds the "global maximum" with telemetry, at best a local sort-of maximum. To find what's best, you need understanding, which you never get from telemetry. Telemetry tells you what was done, not why or what was in the people's mind when it was done.

junon 2 hours ago | parent | prev [-]

This. If I'm forced to use a feature I hate because it's the only way to do something, the "ground truth" reflects that I like that feature. It doesn't tell the whole story.

groby_b an hour ago | parent [-]

Most metrics teams are reasonably competent and are aware of that. Excepting "growth hackers"

I haven't been in a single metrics discussion where we didn't talk about what we're actually measuring, if it reflects what we want to measure, and how to counterbalance metrics sufficiently so we don't build yet another growthhacking disaster.

Doesn't mean that metrics are perfect - they are in fact aggravatingly imprecise - but the ground truth is usually somewhat better than "you clicked it, musta liked it!"

embedding-shape 2 hours ago | parent | prev [-]

> Usage data is the ground truth.

For what, precisely? As far as I know, you can use it to know "how much is X used" but not more than that, and it's not a "ground truth" for anything besides that.

acedTrex 4 hours ago | parent | prev [-]

So if you don't want to spend the time doing that, or as is more accurate in corporate settings, the general turnover of the team is high enough that no one is around long enough to build that deep foundational product knowledge, and to be frank most people do not care enough.

This is why telemetry happens, its faster, easier and more resilient to organizational turmoil.

embedding-shape 3 hours ago | parent [-]

> This is why telemetry happens, its faster, easier and more resilient to organizational turmoil.

I don't disagree with that, I was mainly talking about trying to deliver an experience that makes sense, is intuitive and as helpful and useful as possible, even in exchange for it taking longer time.

Of course this isn't applicable in every case, sometimes you need different tradeoffs, that's OK too. But that some favor quality over shorter implementation time shouldn't drive people crazy, it's just making different tradeoffs.

acedTrex 3 hours ago | parent [-]

> even in exchange for it taking longer time.

I think in terms of corporate teams this is the issue a lot of times, people just are not on the team long enough to build that knowledge. Between the constant reorgs, these days layoffs and other churn the no one puts in the years required to gain the implicit knowledge. So orgs reach for the "tenure independent knowledge base.

6r17 22 minutes ago | parent | prev | next [-]

"You’d think telemetry is screen recording your every move" - that's literally what tracing and telemetry is about.

"Sure, you can spend the weeks to months of expensive and time consuming work it takes to get a fuzzy, half accurate and biased picture of what your users workflows look like through user interviews and surveys. Or you can look at the analytics, which tell you everything you need to know immediately, always up to date, with perfect precision." -> your analytics will never show what you didn't measure - it will only show what you already worked on - at best, it's some kind of validator mechanism - not a driver for feature exploration.

This kind of monitoring need to go through the documented data exposure - and it's a sufficient argument for a company to stop using github immediately if they take security seriously.

But I'd add that if you take security seriously you are not on Github anyway.

Lammy an hour ago | parent | prev | next [-]

> and sending it to the government

It literally is. The network itself is always listening: https://en.wikipedia.org/wiki/Room_641A

The mere act of making a network connection leaks my physical location, the time I'm using my computer, and the fact that I use a particular piece of software. Given enough telemetry endpoints creates a fingerprint unique to me, because it is very unlikely that any other person at the same physical location uses the exact same set of software that I do, almost all of which want to phone home all the goddamn time. It's the metadata that's important here, so payload contents (including encryption) don't even matter.

sdevonoes 3 hours ago | parent | prev | next [-]

Telemetry is the previous obvious step to surveillance. Not the telemetry you implement in your own small bus, but at the scale of microsoft, apple, meta… yeah

ambicapter 3 hours ago | parent | prev | next [-]

> with perfect precision.

Precision isn't accuracy and all that.

3 hours ago | parent | prev [-]
[deleted]
Arch485 4 hours ago | parent | prev | next [-]

Exactly - purely "data driven" decisions are how we end up with ads really close to (or overlapping with) some button you want to press, because the data says that increase click-through rate! But it's actually a user-hostile feature that everyone hates.

mgfist 3 hours ago | parent | next [-]

The reason that feature gets implemented is not because the devs think users will like it ... they know users don't want it, but it drives revenue and pays salaries.

PhoenixFlame101 3 hours ago | parent | prev [-]

But collecting data and looking for insights doesn't mean you mechanically optimize features, especially user-hostile ones? This is just as, if not more, likely to happen when basing your decisions on what people say they want over what they actually do.

defmacr0 3 hours ago | parent [-]

If we were perfectly rational, then yeah, more data should never lead to worse decisions. However, it's easy to fall into the trap where being data-driven makes you only work on those things that you know how to measure.

staticassertion an hour ago | parent | prev | next [-]

It's sort of hilarious to compare "talking to people" with analytics. I'm not defending Github here, but you can't possibly think that "talking to 1M customers" is viable.

almostjazz 42 minutes ago | parent [-]

You could survey a representative sample

staticassertion 40 minutes ago | parent [-]

Not really. (a) People hate responding to surveys and hate emails, you're more likely to lose users than to get data (b) there's no way you're surveying people's in a way that gets you information like "time spent on a page" or "time between commits" or whatever.

This is just nonsense tbh. Surveys and customer outreach solve completely different problems from analytics.

Sytten 4 hours ago | parent | prev [-]

We do both and they yield different learnings. They are complementary. We also have an issue tracking board with upvotes. I would say to your point that you can't improve what you don't measure.

bfivyvysj 4 hours ago | parent [-]

I would say to your point that you can't not spy on me while also spying on me. Maybe just don't?

lukevp 4 hours ago | parent [-]

If I was running a physical business and I wrote down each person’s name and credit card number and the exact time and order they placed, that would be pretty invasive and “spying”. If I write down how many units I sold of each item per day, and the volume of transactions by credit card vs cash, it’s anonymized and I don’t think this would generally be considered “spying”, just normal business metrics. How’s the latter much different than anonymized product analytics?

lamasery 4 hours ago | parent [-]

Watching me use my computer in my house or office is spying.

Aggregating request statistics server-side unless you're only generating those requests to spy on what I'm doing on my computer is more like the not-spying you're talking about.

mgfist 3 hours ago | parent | next [-]

Most telemetry is more along the lines of "user spent N minutes on platform, clicked on these things, looked at these other things" etc etc. And the primary way devs use this data is by aggregating across all users and running a/b tests or viewing longer term trends.

Are some companies spying on you the way you say? Yea, probably. Most of us just want data to know what's working and what's not.

vlovich123 3 hours ago | parent | prev | next [-]

The logical conclusion is you’re asking for no local products and everything to run server side. It’s kind of a ridiculous position that doesn’t change the spying being done other than it’s on the other side of a browser.

lamasery 3 hours ago | parent [-]

I accounted for this in my post. Obviously if you’re making requests just so you can spy, that’s spying.

embedding-shape 3 hours ago | parent | prev [-]

> Watching me use my computer in my house or office is spying.

I agree, but once you cross the borders out to the internet, I'd say you need to stop seeing that as "Me sitting at my computer at home", because you're actually "on someone else's property" at that point essentially. And I say this as someone who care greatly about preserving personal privacy.

lamasery 3 hours ago | parent [-]

I deeply hate that this attitude took over even among “hackers”.

Watching people move their mouse and click stuff on “your webpage” is fucking spying. It’s in my browser. On my machine. Not running on your hardware.

Tracking what I do on my own computer doesn’t stop being spying because the program I’m doing stuff in can make network requests. WTF.

embedding-shape 3 hours ago | parent [-]

> Watching people move their mouse and click stuff on “your webpage” is fucking spying. It’s in my browser. On my machine. Not running on your hardware.

Well, I was mainly talking about network requests, which are quite literally served by "my hardware" when your client reaches out to my servers, and they agree to serve your client. I do agree that it sucks that browser viewports now also are considered "mine" from the perspective of servers, but you do have a choice to execute that code or not, you can always say no.

I don't think it's as much "this attitude took over", people saying that the internet is the wild west and warning you "browse at your own peril" has been around for as long as I can remember.

lamasery 2 hours ago | parent [-]

Yeah server logs don’t bother me. I’m requesting a resource, you unavoidably see that happen.

The attitude that’s changed is that in the 90s and 00s a program that sent information about what you’re doing that wasn’t necessary and expected for how it operates would have been instantly, popularly, and unequivocally labeled spyware by a programmer crowd. Now it’s normal and you get a bunch of folks claiming it’s ok.

nkrisc 4 hours ago | parent | prev | next [-]

> The difference between what people tell you when asked directly and how they actually use your software is actually shocking.

And the difference between what they do and what they want is equally shocking. If what they want isn’t in your app, they can’t do it and it won’t show up in your data.

Quantitative data doesn’t tell you what your users want or care about. It tells you only what they are doing. You can get similar data without spying on your users.

I don’t necessarily think all data gathering is equivalent to spying, but if it’s not entirely opt-in, I think it is effectively spying no matter what you’re collecting, varying only along a dimension of invasiveness.

DrScientist 4 hours ago | parent [-]

> If what they want isn’t in your app, they can’t do it and it won’t show up in your data.

Excellent point.

> but if it’s not entirely opt-in, I think it is effectively spying no matter what you’re collecting, varying only along a dimension of invasiveness.

Every web page visit is logged on the http server, and that's been the default since the mid 1990's. Is that spying?

nkrisc 4 hours ago | parent [-]

In principle, yes, I believe it is a form of spying. Not particularly invasive nor harmful, but spying nonetheless.

Logging every page visited is not a technical requirement of serving the requested resource.

vlovich123 3 hours ago | parent [-]

> Logging every page visited is not a technical requirement of serving the requested resource.

How will you know which page is having problems being served or is having performance problems?

nkrisc 3 hours ago | parent [-]

You won’t, but that’s not what was asked.

Logging the requested resource is not a technical requirement of serving that resource.

jubilanti 4 hours ago | parent | prev | next [-]

Wow, it really is sad how literally unthinkable it is to you and so much of the industry that you could actually talk to your users and customers like human beings instead of just data points.

And you know what happens when you reach out to talk to your customers like human beings instead of spying on them like animals? They like you more and they raise issues that your telemetry would never even think to measure.

It's called user research and client relationship management.

vlovich123 3 hours ago | parent | next [-]

I think you’re overlooking that they were talking about stated and revealed preferences, a well known economic challenge where what people say is important to them and what shows up in the data is a gap. Of course you talk to users and do relationship management. That doesn’t negate the need to understand revealed preferences.

In the OSS world this is not a huge deal. You get some community that’s underserved by the product (ie software package) and they fork, modify, or build something else. If it turned out to be valuable, then you get the old solution complemented or replaced. In the business world this is an existential threat to the business - you want to make sure your users aren’t better served by a competitor who’s focusing on your blindspot.

alexchantavy 3 hours ago | parent | prev | next [-]

The problem they're trying to solve is to find out what functions of their software are most useful for people and what to invest in, and to make directions on product direction.

Yes, vendors can, do, and should talk to users, but then a lot of users don't like receiving cold messages from vendors (and some users go so far as to say that cold messages should _never_ be sent).

So, the alternative is to collect some soft telemetry to get usage metrics. As long as a company is upfront about it and provides an opt-out mechanism, I don't see a problem with it. Software projects (and the businesses around them) die if they don't make the right decisions.

As an open source author and maintainer, I very rarely hear from my users unless I put in the legwork to reach out to them so I completely identify with this.

pc86 3 hours ago | parent [-]

If you have an existing financial relationship with someone it is by definition not a "cold message." People who think they should never, ever be contacted by a company they are paying to use a service of are in the extreme minority. That's "cabin in the woods with no electricity" territory.

chao- 3 hours ago | parent | prev | next [-]

Customer interviews are an indispensable, high-value activity for all businesses. They are a permanent, ongoing capability that the organization must have. A conversation will surface things that analytics will not catch. People will describe their experiences in a qualitative manner that can inspire product improvements that analytics never will.

However, the plural of "anecdote" is not "data". People are unreliable narrators, and you can only ask them so many questions in a limited time amid their busy lives. Also, there are trends which appear sooner in automated analytics by days, weeks, or even months than they would appear in data gathered by the most ambitious interview schedule.

There is a third, middle-ground option as well: surveys. They don't require as much time commitment from the user or the company as a sit-down interview. A larger number of people are willing to engage with them than are willing to schedule a call.

In my experience, all three are indispensable tools.

Sytten 3 hours ago | parent | prev | next [-]

You are inferring your own perception based on my comment, no need to be an asshole here. Like I said elsewhere we do both and they serve different purpose. We also make is very clear and easy to disable in the onboarding. I hope you try to build a business sometimes and open up your perspectives that maybe just maybe you don't have all the answers.

TimorousBestie 2 hours ago | parent [-]

> We also make is very clear and easy to disable in the onboarding.

Yeah, sure. How long is that policy gonna last? How does a user even know that that checkbox does anything?

Once you’ve decided to break a social contract it’s not like you can slap a bandaid on it and it’s all okay now.

> I hope you try to build a business sometimes and open up your perspectives that maybe just maybe you don't have all the answers.

People were building successful businesses long before the Internet.

skeuomorphism 3 hours ago | parent | prev | next [-]

Marketing came to the conclusion that people dont know what they actually want. They decided to lump in engineers and programmers as well, since they started abusing their goodwill.

kalleboo 3 hours ago | parent | prev | next [-]

Apple and Microsoft reached their peak usability when they employed teams of people to literally sit and watch what users did in real life (and listen to them narrating what they want to do), take notes, and ask followup questions.

Everything went to crap in the metric-based era that followed.

3 hours ago | parent | prev | next [-]
[deleted]
7bit an hour ago | parent | prev [-]

Get off your high horse.

Talking to users when you have hundreds of customers does no more than give you an idea of what those specific people need. If you have hundreds of users or more, then data is the only thing that reliably tells you these things.

yoyohello13 3 hours ago | parent | prev | next [-]

The totality of Microsoft's products is proof that this is false. If telemetry and analytics actually mattered for usability, every product Microsoft puts out wouldn't be good instead of garbage.

kodablah 4 hours ago | parent | prev | next [-]

> If you dont have analytics you are flying blind

More like flying based on your knowledge as a pilot and not by the whims of your passengers.

For many CLIs and developer tooling, principled decisions need to reign. Accepting the unquantifiability of usage in a principled product is often difficult for those that are not the target demographic, but for developer tools specifically (be they programming languages, CLIs, APIs, SDKs, etc), cohesion and common sense are usually enough. It also seems real hard for product teams to accept the value of the status quo with these existing, heavily used tools.

mckn1ght an hour ago | parent [-]

Actually it's more like flying in the clouds with no instruments which can lead to spatial disorientation when you exit the cloud cover and realize you're nosediving towards the earth. https://en.wikipedia.org/wiki/Spatial_disorientation

Flying based on the whims of your passengers would be user testing/interviewing, which is a complementary, and IMO necessary, strategy alongside analytics.

ctoth 3 hours ago | parent | prev | next [-]

> If you dont have analytics you are flying blind.

We... we are talking about a CLI tool. A CLI tool that directly uses the API. A tool which already identifies itself with a User-Agent[0].

A tool which obviously knows who is using it. What information are you gathering by running telemetry on my machine that couldn't.. just. be. a. database. query?

Reading the justification the main thing they seem to want to know is if gh is being driven by a human or an agent... Which, F off with your creepy nonsense.

Please don't just use generic "but ma analytics!" when this obviously doesn't apply here?

[0]: https://github.com/cli/cli/blob/3ad29588b8bf9f2390be652f46ee...

ubercore 4 hours ago | parent | prev | next [-]

It makes me think, what `gh` features don't generate some activity in the github API that could as easily guide feature development without adding extra telemetry?

larusso 3 hours ago | parent [-]

Yeah. Unless they plan to move more local git operations in the tool and blur the line between git and gh.

Apylon777 an hour ago | parent | prev | next [-]

How did GitHub ever survive without this telemetry? Was it a web application buried in obscurity?

throwaway27448 4 hours ago | parent | prev | next [-]

I'm pretty ok with the github cli tool team flying blind. The tool isn't exactly a necessary part of any workflow. You don't need telemetry to glean that

merlindru 4 hours ago | parent [-]

that's akin to saying "i do not need their product therefore i don't care"... so what's your point? someone may have made it part of their workflow!

throwaway27448 2 hours ago | parent | next [-]

True. Some people shouldn't use git if their workflow doesn't beg it.

3 hours ago | parent | prev [-]
[deleted]
goosejuice 2 hours ago | parent | prev | next [-]

You could, I don't know, do user interviews with the various customer segments that use your product.

renegade-otter 3 hours ago | parent | prev | next [-]

I agree with you in that regard. That said, knowing that this is Microsoft, the data will be used to extract value from the customers, not provide them with one.

ryandrake 3 hours ago | parent | prev | next [-]

This got me thinking: Are there prominent examples of open source projects that 1. collect telemetry, 2. without a way to opt-out (or obfuscating / making it difficult to opt-out)? This practice seems to be specific to corporate software development.

Why is it that startups and commercial software developers seem to be the only ones obsessed with telemetry? Why do they need it to "optimize user journeys" but open source projects do just fine while flying blind?

theplatman 2 hours ago | parent [-]

open source projects are usually creating something for themselves so it's much easier to know what to build when you are the user

whereas, commercial software has a disconnect between who are the users and developers are

pc86 3 hours ago | parent | prev | next [-]

You can "optimize a successful user journey" by making the software easy to use, making it load so fast people are surprised by it, and talking to your customers. Telemetry doesn't help you do any of that, but it does help you squeeze more money out of them, or find out where you can pop an interstitial ad to goose your ad revenue, and what features you can move up a tier level to increase revenue without providing any additional value.

tomrod 2 hours ago | parent | prev | next [-]

Teams that do this need to just dogfood internally. Once you start collecting telemetry on external users defaulted to opt-in you're not a good faith actor in the ecosystem.

e12e 3 hours ago | parent | prev | next [-]

I think there's room for a distinction between "not using metrics" and "not using data".

Unthinkingly leaning on metrics is likely to help you build a faster, stronger horse, while at the same time avoiding building a car, a bus or a tractor.

a012 3 hours ago | parent | prev [-]

You have all info you need on server side, I don’t believe that you’re totally blind without client tracking