OpenAI, Google and Perplexity don't expose user query analytics. It's difficult to tell if and why your company appears in chat responses.

Turns out the only way you find out currently if you're cited at scale is if you reverse engineer the prompts users might be asking about your brand and track ChatGPT/Gemini/etc. responses for them.

I've built a pipeline, that:

1. crawls your website and your competition

2. analyzes top search keywords from search engines

3. retrieves similar conversations from WildChat (~1M ChatGPT conversations)

4. generates most likely user prompts for your brand.

The result is a set of prompts that users are likely to ask about your company or your products and tracking them shows where LLMs suggest you and where do they reference your competition.

Repo:

https://github.com/syntropicsignal-ai/ai-visibility-audit

I'd love feedback on:

- whether this methodology makes sense

- alternative datasets to WildChat

- better ways to estimate prompt distributions

The 'reverse engineering prompts' approach is interesting, but the variance in LLM responses based on temperature and system prompt updates makes consistency a major hurdle for this type of monitoring. One of the biggest technical challenges is distinguishing between when a model retrieves your site via RAG (live search) versus when it relies on stale training data. In the latter case, you can't really optimize for visibility without a new training cutoff, whereas RAG visibility can be influenced by site structure and indexing. Have you found a way to reliably trigger the search-tool use in your pipeline to ensure you're getting live results? Disclosure: I'm building Sivon HQ, where we track similar AI search visibility metrics.

Current limitation: we don't extract shopping carousels and product suggestions though that's WIP

[flagged]