Crucially, this was never proven in court. And if Apple settle the case it never will be. Let’s think this through. For the accusation to be true, Apple would need to be recording those wake word audio snippets and transmitting them back to their servers for additional processing (likely true), but then they would need to be feeding those snippets in almost real time into a system which forwards them onto advertising partners who then feed that information into targeting networks such that next time you view an ad on your phone the information is available to help select the relevant ad.

Why would Apple do that? Especially given both their brand and reputation as a privacy-first company combined with the large amounts of product design and engineering work they’ve put into preventing apps from doing exactly this kind of thing by enforcing permission-based capabilities and ensuring a “microphone active” icon is available at all times when an app is listening in.

The AI trust crisis 14th December 2023 Dropbox added some new AI features. In the past couple of days these have attracted a firestorm of criticism. Benj Edwards rounds it up in Dropbox spooks users with new AI features that send data to OpenAI when used. The key issue here is that people are worried that their private files on Dropbox are being passed to OpenAI to use as training data for their models—a claim that is strenuously denied by Dropbox. As far as I can tell, Dropbox built some sensible features—summarize on demand, “chat with your data” via Retrieval Augmented Generation—and did a moderately OK job of communicating how they work... but when it comes to data privacy and AI, a “moderately OK job” is a failing grade. Especially if you hold as much of people’s private data as Dropbox does! Two details in particular seem really important. Dropbox have an AI principles document which includes this: Customer trust and the privacy of their data are our foundation. We will not use customer data to train AI models without consent. They also have a checkbox in their settings that looks like this: Update: Some time between me publishing this article and four hours later, that link stopped working. I took that screenshot on my own account. It’s toggled “on”—but I never turned it on myself. Does that mean I’m marked as “consenting” to having my data used to train AI models? I don’t think so: I think this is a combination of confusing wording and the eternal vagueness of what the term “consent” means in a world where everyone agrees to the terms and conditions of everything without reading them. But a LOT of people have come to the conclusion that this means their private data—which they pay Dropbox to protect—is now being funneled into the OpenAI training abyss. People don’t believe OpenAI # Here’s copy from that Dropbox preference box, talking about their “third-party partners”—in this case OpenAI: Your data is never used to train their internal models, and is deleted from third-party servers within 30 days. It’s increasing clear to me like people simply don’t believe OpenAI when they’re told that data won’t be used for training. What’s really going on here is something deeper then: AI is facing a crisis of trust. I quipped on Twitter: “OpenAI are training on every piece of data they see, even when they say they aren’t” is the new “Facebook are showing you ads based on overhearing everything you say through your phone’s microphone” Here’s what I meant by that. Facebook don’t spy on you through your microphone # Have you heard the one about Facebook spying on you through your phone’s microphone and showing you ads based on what you’re talking about? This theory has been floating around for years. From a technical perspective it should be easy to disprove: Mobile phone operating systems don’t allow apps to invisibly access the microphone. Privacy researchers can audit communications between devices and Facebook to confirm if this is happening. Running high quality voice recognition like this at scale is extremely expensive—I had a conversation with a friend who works on server-based machine learning at Apple a few years ago who found the entire idea laughable. The non-technical reasons are even stronger: Facebook say they aren’t doing this. The risk to their reputation if they are caught in a lie is astronomical. As with many conspiracy theories, too many people would have to be “in the loop” and not blow the whistle. Facebook don’t need to do this: there are much, much cheaper and more effective ways to target ads at you than spying through your microphone. These methods have been working incredibly well for years. Facebook gets to show us thousands of ads a year. 99% of those don’t correlate in the slightest to anything we have said out loud. If you keep rolling the dice long enough, eventually a coincidence will strike. Here’s the thing though: none of these arguments matter. If you’ve ever experienced Facebook showing you an ad for something that you were talking about out-loud about moments earlier, you’ve already dismissed everything I just said. You have personally experienced anecdotal evidence which overrides all of my arguments here.

One consistent theme I’ve seen in conversations about this issue is that people are much more comfortable trusting their data to local models that run on their own devices than models hosted in the cloud. The good news is that local models are consistently both increasing in quality and shrinking in size.

my critique of Manjoo’s article specifically and the ongoing privacy hysteria broadly is not simply about definitions or philosophy. It’s about fundamental assumptions. The default state of the Internet is the endless propagation and collection of data: you have to do work to not collect data on one hand, or leave a data trail on the other. This is the exact opposite of how things work in the physical world: there data collection is an explicit positive action, and anonymity the default.

I believe the privacy debate needs to be reset around these three assumptions: Accept that privacy online entails trade-offs; the corollary is that an absolutist approach to privacy is a surefire way to get policy wrong. Keep in mind that the widespread creation and spread of data is inherent to computers and the Internet, and that these qualities have positive as well as negative implications; be wary of what good ideas and positive outcomes are extinguished in the pursuit to stomp out the negative ones. Focus policy on the physical and digital divide. Our behavior online is one thing: we both benefit from the spread of data and should in turn be more wary of those implications. Making what is offline online is quite another.

Even if chatbots were to fix their accuracy issues, Google would still have a business model problem to contend with. The company makes money when people click ads next to search results, and it’s awkward to fit ads into conversational replies. Imagine receiving a response and then immediately getting pitched to go somewhere else — it feels slimy, and unhelpful. Google thus has little incentive to move us beyond traditional search, at least not in a paradigm-shifting way, until it figures out how to make the money aspect work. In the meantime, it’ll stick with the less impressive Google Assistant.

“Google doesn’t inherently want you, at an inherent level, to just get the answer to every problem. Because that might reduce the need to go click around the web, which would then reduce the need for us to go to Google.”

if you go into the Shanghai City Guide, it’s easy to put together a list of must-visit restaurants or family-friendly activities. Plus, not only is it easy to create bookmarks of places you want to save, but you can also browse other public bookmarks curated by creators and influencers, and then sort the bookmarks by location or distance. This type of bookmark discovery is a completely different use case than how we use Google Maps or Yelp today. Our Western location-based platforms use a manual, search-based process that primarily works for high-intent discovery. In contrast, thanks to the algorithms and user behavior on Douyin, the app can push new places to a user that he or she would never have thought to search for. Short video effectively enables the continuous discovery of the best places around you, and it gets better with every swipe as the platform better understands you and your interests.

In other words, when Kylie Jenner posts a petition demanding that Meta “Make Instagram Instagram again”, the honest answer is that changing Instagram is the most Instagram-like behavior possible.

The first trend is the shift towards ever more immersive mediums. Facebook, for example, started with text but exploded with the addition of photos. Instagram started with photos and expanded into video. Gaming was the first to make this progression, and is well into the 3D era. The next step is full immersion — virtual reality — and while the format has yet to penetrate the mainstream this progression in mediums is perhaps the most obvious reason to be bullish about the possibility.

The second trend is the increase in artificial intelligence. I’m using the term colloquially to refer to the overall trend of computers getting smarter and more useful, even if those smarts are a function of simple algorithms, machine learning, or, perhaps someday, something approaching general intelligence.

The third trend is the change in interaction models from user-directed to computer-controlled. The first version of Facebook relied on users clicking on links to visit different profiles; the News Feed changed the interaction model to scrolling. Stories reduced that to tapping, and Reels/TikTok is about swiping. YouTube has gone further than anyone here: Autoplay simply plays the next video without any interaction required at all.

Advertisers don’t really want to know who you are - they want to show diaper ads to people who have babies, not to show them to people who don’t, and to have some sense of which ads drove half a million sales and which ads drove a million sales.

In practice, ‘showing car ads to people who read about cars’ led the adtech industry to build vast piles of semi-random personal data, aggregated, disaggregated, traded, passed around and sometimes just lost, partly because it could and partly because that appeared to be the only way to do it. After half a decade of backlash, there are now a bunch of projects trying to get to the same underlying advertiser aims - to show ads that are relevant, and get some measure of ad effectiveness - while keeping the private data private.

Apple has pursued a very clear theory that analysis and tracking is private if it happens on your device and is not private if leaves your device or happens in the cloud. Hence, it’s built a complex system of tracking and analysis on your iPhone, but is adamant that this is private because the data stays on the device. People have seemed to accept this (so far - or perhaps the just haven’t noticed it), but acting on the same theory Apple also created a CSAM scanning system that it thought was entirely private - ‘it only happens your device!’ - that created a huge privacy backlash, because a bunch of other people think that if your phone is scanning your photos, that isn’t ‘private’ at all. So is ‘on device’ private or not? What’s the rule? What if Apple tried the same model for ‘private’ ads in Safari? How will the public take FLoC? I don’t think we know.

On / off device is one test, but another and much broader is first party / third party: the idea it’s OK for a website to track what you do on that website but not OK for adtech companies to track you across many different websites. This is the core of the cookie question

At this point one answer is to cut across all these questions and say that what really matters is whether you disclose whatever you’re doing and get consent. Steve Jobs liked this argument. But in practice, as we've discovered, ‘get consent’ means endless cookie pop-ups full of endless incomprehensible questions that no normal consumer should be expected to understand, and that just train people to click ‘stop bothering me’. Meanwhile, Apple’s on-device tracking doesn't ask for permission, and opts you in by default, because, of course, Apple thinks that if it's on the device it's private. Perhaps ‘consent’ is not a complete solution after all.

If you can only analyse behaviour within one site but not across many sites, or make it much harder to do that, companies that have a big site where people spend lots of time have better targeting information and make more money from advertising. If you can only track behaviour across lots of different sites if you do it ‘privately’ on the device or in the browser, then the companies that control the device or the browser have much more control over that advertising

it is very difficult to figure out what specific effect ATT has because there are so many factors involved

If ATT were so significantly kneecapping revenue, I would think we would see a pronounced skew against North America compared to elsewhere. But that is not the case. Revenue in North America is only slightly off compared to the company total, and it is increasing how much it earns per North American user compared to the rest of the world.

iOS is far more popular in the U.S. and Canada than it is in Europe, but Meta incurred a greater revenue decline — in absolute terms and, especially, in percentage terms — in Europe. Meta was still posting year-over-year gains in both those regions until this most recent quarter, even though ATT rolled out over a year ago.

there are those who believe highly-targeted advertisements are a fair trade-off because they offer businesses a more accurate means of finding their customers, and the behavioural data collected from all of us is valuable only in the aggregate. That is, as I understand it, the view of analysts like Seufert, Benedict Evans, and Ben Thompson. Frequent readers will not be surprised to know I disagree with this premise. Regardless of how many user agreements we sign and privacy policies we read, we cannot know the full extent of the data economy. Personal information about us is being collected, shared, combined, and repackaged. It may only be profitable in aggregate, but it is useful with finer granularity, so it is unsurprising that it is indefinitely warehoused in detail.

Seufert asked, rhetorically, “what happens when ads aren’t personalized?”, answering “digital ads resemble TV ads: jarring distractions from core content experience. Non-personalized is another way of saying irrelevant, or at best, randomly relevant.”

does it make sense to build the internet’s economy on the backs of a few hundred brokers none of us have heard of, trading and merging our personal information in the hope of generating a slightly better click-through rate?

Then there is the much bigger question of whether people should even be able to opt into such widespread tracking. We simply cannot be informed consumers in every aspect of our lives, and we cannot foresee how this information will be used and abused in the full extent of time. It sounds boring, but what is so wrong with requiring data minimization at every turn, permitting only the most relevant personal data to be collected, and restricting the ability for this information to be shared or combined?

Does ATT really “[deprive] consumers of widespread ad relevancy and advertisers and publishers of commercial opportunity”? Even if it does — which I doubt — has that commercial opportunity really existed with meaningful consumer awareness and choice? Or is this entire market illegitimate, artificially inflated by our inability to avoid becoming its subjects?

Conflicts like these are one of many reasons why privacy rights should be established by regulators, not individual companies. Privacy must not be a luxury good, or something you opt into, and it should not be a radical position to say so. We all value different degrees of privacy, but it should not be possible for businesses to be built on whether we have rights at all. The digital economy should not be built on such rickety and obviously flawed foundations.