MSC3911: Linking media to events #3911

richvdh · 2022-10-16T22:38:17Z

proposals/3911-linking-media-to-events.md

uhoreg · 2022-10-17T19:25:17Z

proposals/3911-linking-media-to-events.md

+
+### Overview
+
+After an item of media is uploaded, it must be linked to an event (via


For MLS, we may need to reference media within a to-device event, but we might be able to link it to an in-room event. I'll have to look into this more, but flagging this as something that will need to be considered.

there's also ephemeral-ish things like user profiles which aren't directly related to events, though are duplicated/copied to membership events in most cases.

proposals/3911-linking-media-to-events.md

Munna231

#

richvdh · 2022-10-23T23:16:18Z

I've moved a bunch of the changes here out to ~~MSC3716~~ MSC3916, in the hope of making them a little more digestible.

deepbluev7 · 2022-10-23T23:19:35Z

I've moved a bunch of the changes here out to MSC3716, in the hope of making them a little more digestible.

You meant MSC3916 (if anyone else has issues following that link :D)

richvdh · 2022-10-23T23:20:58Z

I've moved a bunch of the changes here out to MSC3716, in the hope of making them a little more digestible.

You meant MSC3916 (if anyone else has issues following that link :D)

Thank you, yes I did 🤦. Edited.

reivilibre · 2023-03-10T17:08:07Z

proposals/3911-linking-media-to-events.md

+  In addition, this could cause duplication of media in the remote media cache,
+  if the implementation does not take steps to deduplicate (eg, storing media
+  by content hash rather than media id).


Can we spec a response header for the media download that provides a suitable hash of the object?
e.g. X-Matrix-Media-Hash: blake2,AAAAAAAAAAAAAAAAAAAAAAAAAAA....
That way, a homeserver (or client!) which is downloading a copy of the media can abort the download if it finds the hash already in its media store. This means that copies of large media do not lead to each server downloading it for each copy.
(Further, for bandwidth-sensitive clients, a HEAD could be used to get the hash before even initiating the download?)

De-duplication is already possible at an implementation level, and would probably be handled by a different MSC.

De-duplication is already possible at an implementation level

I think that reivilibre wasn't talking about the way clients/homeservers are deduplicating files after download, but rather before they're downloaded. I agree that former is an implementation detail that doesn't require uniformity across software, and as such doesn't need to be covered in protocol spec. But if the intention here is to avoid the very process of downloading a file multiple times (from a different linked MXC each time), can this be done without protocol declaring one well known way to discover e.g. that hash of underlying file by HEAD request?

I'm specifically thinking of a situation where you've got a user looking at sticker selector and not-so-maliciously spamming them by clicking them 100s of times. Some people do that, it isn't really a bad behavior in some rooms and contexts (slightly distant example would be custom emote spam on Twitch chats). Since the stickers sent in each event are no longer sharing the same MXC, AFAIK client (and homeserver, if sender is remote) will have to download the sticker 100s of times, and thus generate a lot of unnecessary traffic. Only then it will be able to deduplicate file for local storage/cache.

So my (and I assume reivilibre's) concern here is that with the proposal as it is right now, there's no way for client/homeserver to avoid that "traffic duplication", so to speak. Or is there? 😅

Hm, on the other hand, that choice of hash type used for deduplication before download, might eventually have an effect on future implementations of dedupe after download. For example, iirc currently in Synapse there's a mapping of MXC URI to local filename which is a random string. With this MSC, each server would have to store a mapping from the new generated IDs to the local file (in Synapse's case, again, a random name). If protocol adds a way to map MXC IDs to file hash early, is there a reason for servers to not store files by naming them after their hashes (as received over e.g. federation)? This would save you one more lookup: MXC IDs -> file hash -> randomized local name.

Oh, and I haven't thought how/if this "early dedupe" could realistically work with files attached to E2EE events, and whether it would add any additional metadata leak other than what's caused by this MSC as is.

richvdh · 2023-04-25T14:05:07Z

proposals/3911-linking-media-to-events.md

+1. A new "media upload" endpoint is defined, `POST
+   /_matrix/client/v1/media/upload`.  It is based on the existing
+   [`/_matrix/media/v3/upload`](https://spec.matrix.org/v1.4/client-server-api/#post_matrixmediav3upload)
+   endpoint, but media uploaded this way is not initially viewable (except to
+   the user that uploaded it). This is referred to as a "restricted" media item.


TODO: extend this to /_matrix/media/v1/create, as added by MSC2246.

turt2live · 2023-09-18T03:58:54Z

proposals/3911-linking-media-to-events.md

+   The new media item can be attached to a new event, and generally functions
+   in every way the same as uploading a brand new media item.


Can the user copy restricted media they don't have access to? (I assume not, but what is the appropriate error code?)

I'd say no, it's the same as /download and /thumbnail. 403 and M_UNAUTHORIZED.

turt2live · 2023-09-18T04:01:33Z

proposals/3911-linking-media-to-events.md

+removed. Currently, Synapse does not support removing the events (see also
+[synapse#4720](https://github.com/matrix-org/synapse/issues/4720)); but if at


Synapse was fixed a few days ago, apparently :)

turt2live · 2023-09-18T04:08:26Z

proposals/3911-linking-media-to-events.md

+   It is expected that servers will continue to treat such media as unrestricted
+   (at least for local users), but it would be legitimate for them to, for example,
+   return a different `mxc:` URI for each requesting user, and only allow each user
+   access to the corresponding `mxc:` URI.


I'm not sure I see the advantage of a server issuing distinct URIs for previews. Is there a motivation for putting this here?

I think I was trying to reason about how we might apply access restrictions to media that is part of a URL preview (if we decided that was a sensible thing to do). It may well be that the extra text confuses more than it clarifies.

leaving url previews as a way to keep "uploading" unrestricted media would leave open the attack vector this MSC is trying to close.

turt2live · 2023-09-18T04:09:10Z

proposals/3911-linking-media-to-events.md

+
+## Unstable prefix
+
+TODO


(this is what MMR will be using)

turt2live · 2023-09-18T04:32:48Z

proposals/3911-linking-media-to-events.md

+     If any of the `attach_media` parameters do not correspond to known
+     restricted media items, or they refer to restricted media items that have
+     already been attached, the server responds with a 400 error with
+     `M_INVALID_PARAM`.


Should there also be a limitation that one can only attach media that they uploaded?

erm... good question.

I think the answer is "yes, there should", to guard against the possibility that someone gets hold of the mxc: URI between it being uploaded and the media being attached to an event. (In which case, they could attach it to their own event and get access to the media.) Particularly in the case of /create rather than /upload, in which the mxc: URI would appear in access logs.

richvdh · 2023-11-14T16:21:32Z

proposals/3911-linking-media-to-events.md

+   This "copy" api is to be used by clients when forwarding events with media
+   attachments.


We'd need to use this for m.sticker events too.

(and custom emoji, when that exists)

turt2live · 2023-11-14T16:27:10Z

proposals/3911-linking-media-to-events.md

+   --gc0p4Jq0M2Yt08jU534c0p
+   ```
+
+5. New "media copy" API


A downside of the media copy API is clients would no longer be able to reliably cache media. For the example of stickers or emoji, popular/busy rooms could have tens of references to the same media object, which causes tens of downloads because the client doesn't know different.

Possible alternatives (for future MSCs, imo) would be:

Allow some media objects to be reference counted/linked to multiple events.

Include a hash of the file in the event, so clients can locally realize that the objects are all the same content-wise.

I'm more preferential to the second.

turt2live · 2023-11-14T16:28:58Z

proposals/3911-linking-media-to-events.md

+   --gc0p4Jq0M2Yt08jU534c0p
+   ```
+
+5. New "media copy" API


going down the rabbit hole of custom emoji, there's an interaction with MSC4027 we will have to figure out.

turt2live · 2023-12-18T07:01:23Z

proposals/3911-linking-media-to-events.md

+
+Fixes [synapse#1263](https://github.com/matrix-org/synapse/issues/1263).
+
+## Potential issues


@AndrewRyanChama says:

How does this interact with m.replace? Will the client be expected to /copy all of the medias or will it be handled automagically?

AndrewRyanChama · 2023-12-23T20:11:21Z

proposals/3911-linking-media-to-events.md

+#### Redacting events
+
+Under this proposal, servers can determine which media is referenced by an
+event that is redacted, and add that media to a list to be cleaned up.


The problem with this is that if the user who sent the media leaves the room, the homeserver may no longer get the reaction events. That means there will be no mechanism for those events to be removed.

AndrewRyanChama · 2023-12-23T20:18:21Z

proposals/3911-linking-media-to-events.md

+   Content-Type: application/json
+
+   { "restrictions": {
+       "event_id": "$Rqnc-F-dvnEYJTyHq_iKxU2bZ1CI92-kuZq3a5lr5Zg"


This means that the association between (encrypted) media and event ids will be plainly accessible to the homeserver, and federated servers. I'd imagine that isn't something we actually care about but something that others may complain about.

Given that /copy presumably can't actually decrypt the media, that also means that the homeserver can associate any copied encrypted media as well

AndrewRyanChama · 2023-12-23T20:49:59Z

proposals/3911-linking-media-to-events.md

+
+After an item of media is uploaded, it must be linked to an event (via
+parameters to the `/send` api). A given piece of media is only visible
+to a user if the user can see the corresponding event.


Since reference counting is required anyways, I feel like it's better if the server does it automatically. I wrote a short doc on that here: #4086

Kladki · 2024-05-28T18:03:29Z

proposals/3911-linking-media-to-events.md

+   The `/_matrix/federation/v1/media/download` and `/_matrix/federation/v1/media/thumbnail`
+   endpoints specified by [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916)
+   are extended: the returned json object may have a property `restrictions`.


MSC3916 doesn't specify a federation /thumbnail endpoint.

there are several updates required to this MSC - the priority has been to get 3916 into working order, then the team will move onto this MSC afterwards.

MSC3916 was merged 🥳

HarHarLinks

reviewing with my Nordeck hat on.

HarHarLinks · 2024-08-13T15:21:27Z

proposals/3911-linking-media-to-events.md

+   The existing endpoint is deprecated. Media uploaded via the existing endpoint
+   is "unrestricted".
+
+2. Attaching media


the proposal is called "Linking media to events" and "linking" terminology is used outside of this list item. is this intentional? can we agree on a single term otherwise? "attaching" seems better suited.

HarHarLinks · 2024-08-13T15:21:29Z

proposals/3911-linking-media-to-events.md

+
+     If any of the `attach_media` parameters do not correspond to known
+     restricted media items, or they refer to restricted media items that have
+     already been attached, the server responds with a 400 error with


reading this proposal top down, it is not entirely clear whether "already attached" means a different event or a duplicate query parameter.

HarHarLinks · 2024-08-13T15:27:01Z

proposals/3911-linking-media-to-events.md

+   If the media is not attached to either an event or a profile within a reasonable period
+   (say, ten minutes), then the server is free to assume that the user has changed their


for clarity: the period starts as the media upload has finished, which is particularly relevant with async uploads?

HarHarLinks · 2024-08-13T15:28:13Z

proposals/3911-linking-media-to-events.md

+
+   The new `/download` and `/thumbnail` endpoints added in
+   [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916) are
+   updated the server must check if the requesting user or server has


Suggested change

updated the server must check if the requesting user or server has

updated such that the server must check if the requesting user or server has

HarHarLinks · 2024-08-13T15:30:47Z

proposals/3911-linking-media-to-events.md

+   cache the restrictions list.
+
+   If neither `event_id` nor `profile_user_id` are present, the requesting
+   user should assume that an unknown restriction is present, and not allow access


Suggested change

user should assume that an unknown restriction is present, and not allow access

homeserver should assume that an unknown restriction is present, and not allow access

HarHarLinks · 2024-08-13T15:49:36Z

proposals/3911-linking-media-to-events.md

+* The only protection for media is the obscurity of the URL, and URLs are
+  easily leaked (eg accidental sharing, access
+  logs). [synapse#2150](https://github.com/matrix-org/synapse/issues/2150)
+* Anybody (including non-matrix users) can cause a homeserver to copy media


since authenticated media landed, I think

Suggested change

* Anybody (including non-matrix users) can cause a homeserver to copy media

* Any locally authenticated user can cause a homeserver to copy media

HarHarLinks · 2024-08-13T15:54:39Z

proposals/3911-linking-media-to-events.md

+   It is expected that servers will continue to treat such media as unrestricted
+   (at least for local users), but it would be legitimate for them to, for example,
+   return a different `mxc:` URI for each requesting user, and only allow each user
+   access to the corresponding `mxc:` URI.


leaving url previews as a way to keep "uploading" unrestricted media would leave open the attack vector this MSC is trying to close.

HarHarLinks · 2024-08-13T15:58:58Z

proposals/3911-linking-media-to-events.md

+ * special-casing the bridge AS user to permit it to upload media without
+   expiry.


Suggested change

* special-casing the bridge AS user to permit it to upload media without

expiry.

* special-casing the bridge AS user to permit it to upload unrestricted media.

other ideas:

attaching media to messages retroactively (e.g. the pastebin copy to the original long message)

tracking media origin room and/or user so AS don't upload entirely unrestricted media

HarHarLinks · 2024-08-16T09:25:04Z

proposals/3911-linking-media-to-events.md

+
+Fixes [synapse#1263](https://github.com/matrix-org/synapse/issues/1263).
+
+## Potential issues


please consider that widgets also use media, i.e. #4039

to summarize, linking media is not a big issue for events of a well-known structure referencing them - with regards to widgets, that means that clients can parse the payload and handle it automatically. however especially for widgets it is often attractive to send nonstandard events (let alone ones with some custom encoding), that prohibit this.

there can be use cases (with widgets or without), where multiple media are attached to an event that represents a certain state at a point in time and which is edited collaboratively. however editing would usually involve changing only one of the referenced media items at most, resulting in an immense overhead when having to copy all the other attached media as well when sending an update (example implementation).

MSC3911: Linking media to events

b1da78c

minor edits

58e6150

richvdh commented Oct 16, 2022

View reviewed changes

proposals/3911-linking-media-to-events.md Outdated Show resolved Hide resolved

ShadowJonathan reviewed Oct 17, 2022

View reviewed changes

proposals/3911-linking-media-to-events.md Show resolved Hide resolved

ShadowJonathan reviewed Oct 17, 2022

View reviewed changes

proposals/3911-linking-media-to-events.md Show resolved Hide resolved

deepbluev7 reviewed Oct 17, 2022

View reviewed changes

proposals/3911-linking-media-to-events.md Show resolved Hide resolved

uhoreg reviewed Oct 17, 2022

View reviewed changes

Half-Shot reviewed Oct 18, 2022

View reviewed changes

proposals/3911-linking-media-to-events.md Outdated Show resolved Hide resolved

richvdh commented Oct 18, 2022

View reviewed changes

proposals/3911-linking-media-to-events.md Outdated Show resolved Hide resolved

turt2live self-requested a review October 18, 2022 17:52

Munna231 suggested changes Oct 19, 2022

View reviewed changes

Move parts of this out to MSC3916

cc853ef

richvdh force-pushed the rav/proposal/linking_media_to_events branch from 464e5a4 to cc853ef Compare October 23, 2022 23:17

richvdh closed this Oct 23, 2022

richvdh reopened this Oct 23, 2022

add some clarifications

4a10395

cvwright mentioned this pull request Nov 10, 2022

MSC1767: Extensible event types & fallback in Matrix (v2) #1767

Merged

reivilibre reviewed Mar 10, 2023

View reviewed changes

spaetz mentioned this pull request Apr 18, 2023

MSC3860: Media Download Redirects #3860

Merged

richvdh commented Apr 25, 2023

View reviewed changes

turt2live mentioned this pull request Jul 17, 2023

Implement new media auth proposals (MSC3916 & MSC3911) t2bot/matrix-media-repo#421

Open

turt2live mentioned this pull request Sep 10, 2023

Media authentication & event linking (MSC3916, MSC3911) t2bot/matrix-media-repo#465

Draft

13 tasks

turt2live reviewed Sep 18, 2023

View reviewed changes

richvdh commented Nov 14, 2023

View reviewed changes

turt2live reviewed Nov 14, 2023

View reviewed changes

turt2live mentioned this pull request Nov 27, 2023

MSC discussion: Custom emoji/stickers and interactions with authenticated & linked media matrix-org/matrix-spec#1679

Open

This was referenced Dec 16, 2023

MSC2704: Explicitly allow alternative origins in MXC URIs and specify deduplication requirements on uploads #2704

Closed

MSC3468: MXC to Hashes #3468

Open

This comment was marked as duplicate.

Sign in to view

turt2live reviewed Dec 18, 2023

View reviewed changes

AndrewRyanChama reviewed Dec 23, 2023

View reviewed changes

This was referenced Jan 3, 2024

Delete media from the server when the event is redacted element-hq/element-meta#398

Open

There is no way to delete content from the media repo matrix-org/matrix-spec#226

Open

Auth for content repo (and enforcing GDPR erasure) #3796

Open

Kladki reviewed May 28, 2024

View reviewed changes

Kladki mentioned this pull request Jun 25, 2024

client-api: Add support for linking media to events ruma/ruma#1869

Draft

HarHarLinks suggested changes Aug 16, 2024

View reviewed changes

ara4n mentioned this pull request Sep 3, 2024

MSC2278: Deleting attachments for expired and redacted messages #2278

Open

f0x52 mentioned this pull request Sep 18, 2024

Federated media multipart/mixed JSON object shouldn't be required to be empty matrix-org/matrix-spec#1952

Open

anoadragon453 mentioned this pull request Oct 2, 2024

Purge Media of banned users and moderated messages element-hq/synapse#17777

Closed


		### Overview

		After an item of media is uploaded, it must be linked to an event (via

		The new media item can be attached to a new event, and generally functions
		in every way the same as uploading a brand new media item.

		removed. Currently, Synapse does not support removing the events (see also
		[synapse#4720](https://github.com/matrix-org/synapse/issues/4720)); but if at

-TODO
+While this MSC is not considered stable, implementations should use the following mapped values.
+| Stable | Unstable |
+|-|-|
+| `/_matrix/client/v1/media/upload` | `/_matrix/client/unstable/org.matrix.msc3911/media/upload` |
+| `/_matrix/client/v1/media/create` | `/_matrix/client/unstable/org.matrix.msc3911/media/create` |
+| `attach_media` | `org.matrix.msc3911.attach_media` |
+| `restrictions` | `org.matrix.msc3911.restrictions` (`event_id` and `profile_user_id` are not prefixed) |
+| `/_matrix/client/v1/media/copy/:serverName/:mediaId` | `/_matrix/client/unstable/org.matrix.msc3911/media/copy/:serverName/:mediaId` |

		This "copy" api is to be used by clients when forwarding events with media
		attachments.


		Fixes [synapse#1263](https://github.com/matrix-org/synapse/issues/1263).

		## Potential issues

		If the media is not attached to either an event or a profile within a reasonable period
		(say, ten minutes), then the server is free to assume that the user has changed their

	updated the server must check if the requesting user or server has
	updated such that the server must check if the requesting user or server has

	user should assume that an unknown restriction is present, and not allow access
	homeserver should assume that an unknown restriction is present, and not allow access

	* Anybody (including non-matrix users) can cause a homeserver to copy media
	* Any locally authenticated user can cause a homeserver to copy media

		* special-casing the bridge AS user to permit it to upload media without
		expiry.

MSC3911: Linking media to events #3911

Are you sure you want to change the base?

MSC3911: Linking media to events #3911

Conversation

richvdh commented Oct 16, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Munna231 left a comment

Choose a reason for hiding this comment

richvdh commented Oct 23, 2022 • edited Loading

deepbluev7 commented Oct 23, 2022 • edited Loading

richvdh commented Oct 23, 2022

reivilibre Mar 10, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rom4nik Jan 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turt2live Sep 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

turt2live Sep 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as duplicate.

Choose a reason for hiding this comment

AndrewRyanChama Dec 23, 2023 • edited Loading

Choose a reason for hiding this comment

AndrewRyanChama Dec 23, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HarHarLinks left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richvdh commented Oct 16, 2022 •

edited

Loading

richvdh commented Oct 23, 2022 •

edited

Loading

deepbluev7 commented Oct 23, 2022 •

edited

Loading

reivilibre Mar 10, 2023 •

edited

Loading

rom4nik Jan 28, 2024 •

edited

Loading

turt2live Sep 18, 2023 •

edited

Loading

turt2live Sep 18, 2023 •

edited

Loading

AndrewRyanChama Dec 23, 2023 •

edited

Loading

AndrewRyanChama Dec 23, 2023 •

edited

Loading