If youāve been looking for a way to have you own Matrix homeserver without having to run it yourself, you may be interested to hear that Modular (the Matrix hosting provider run by New Vector, the startup which hires many of the Matrix core team) is now offering a personal-sized small homeserver hosting service, supporting a minimum size of 5 user servers.
A lot of recent performance work on Synapse has been driven by the need to make smaller dedicated servers more efficient to run - and so if you run your own homeserver youāll be benefiting from all this work too :) Meanwhile, if you choose to outsource your server hosting to Modular, youāll be indirectly supporting core Matrix and Synapse development, given most of the core Matrix team work for New Vector - itās through buying services like this which lets us keep folks able to hack on Matrix as their day job.
Nice! Manu and Benoit, who work on the Riot Mobile, discuss the development status of RiotX.
Dept of Servers š¢
Synapse
A bit of everything this week, weāve made changes to support the upcoming edits and reactions release, worked on soft log out, experimented with improving general perf for small homeservers, landed open tracing support, improved db query load.
Next week weāll see about landing the small homeserver perf improvements, work on id hashing in sydent, fix some e2ee bugs (made easier to track down with OpenTracing), do some more database performance work and start gradually rolling out the new Sygnal instance.
Construct
Jason back on it:
This week in Matrix, Construct made the crazy-loading mode of client sync the default. Crazy-loading is an approach to initial sync that goes beyond lazy-loading for a better UX. Itās even backwards compatible with clients that donāt support lazy-loading.
Construct also made significant progress on implementing version 3 and 4 rooms during the week. This is nearly complete, and should be ready for testing by the weekend.
Good to know there is progress with new room versions as more and more rooms start to be moved over to v4. #zemos-test:matrix.org for testing and more info.
Dendrite
This week weāve implemented profile retrieval over federation, single event retrieval, room tagging as well as host of bug fixes.
Next week weāll be looking at state resolution and implementing our latest and greatest algorithm needed by modern room versions.
Work continues on the major revamp of ruma-events mentioned in the last update.
ā¦
There are also a few modules that are somewhat blocked on an issue in ring. Some of the types in ruma-events contain types from ruma-signatures which donāt implement Clone and PartialEq because they contain types from ring which donāt.
ā¦
Rust 1.36 was released, and it includes stabilization of the Future trait, one of the long-awaited building blocks for first-class async support in Rust. [ā¦] the biggest reason for Rumaās development hiatus is waiting for async networking in Rust to mature, and this is one of the final pieces of foundational support weāve been waiting for. The remaining pieces are async/await syntax, which is expected in either the next version or the one following it, and finally, waiting for important libraries like Hyper and Tokio, as well as web frameworks, to adopt the new stuff.
Thanks to Black Hat, libQuotient gained support of .well-known - a very useful feature to connect to Modular-hosted homeservers!
Also, the first block of E2EE functionality from aa13q has been merged to libQuotient master - so far itās just uploading the keys but receiving messages is already well in the works!
mautrix-telegram v0.6.0 was released. Recent changes include bridging strikethrough, underline and nested formatting to telegram and some bug fixes, including one security fix. Full changelog on GitHub.
Debian 10 was also released recently, which means v0.6 is the last version with Python 3.5 support. Starting from v0.7.0, mautrix-telegram will only support Python 3.6 and up.
mautrix-telegram v0.6.0 also includes Native Matrix edit support, message editing between platforms.
Hi folks, the slack bridge has had another RC release this week 0.3.0-rc2 which has been deployed onto matrix.org :). In other news, we are nearly done with the port of the bridge to Typescript (slated for the 0.4 release) which has allowed us to clean up the codebase significantly and splat a lot of bugs.
Iām for any movement toward TypeScript - seems to be a winner in the JS-world. Says Half-Shot:
Iām a bit fanatical about Typescript, itās objectively better to write things in TS than JS if you have the freedom to do so. Itās also allowed us to keep the bug count down on the Discord bridge, so Iām starting to look at the other bridges for typescript support too.
The new Spec proposal MSC2162: Signaling Errors at Bridges landed! It is about adding permanent errors: The ability of bridges to mark events as not delivered to all participants. While there is already code supporting the feature, the Spec process is important for getting everyone on board and finding potential problems with the current approach.
In spite of being a relatively small proposal, there were already a lot of suggestions and directions in which it can evolve. Shoutout and thanks to everyone who already contributed to it with their comments!
Meanwhile on the more practical front a fork of Riot Web was extended to now support the actual visual display of bridge error markings on messages.
Spectral supports .well-known now obviously. see libQuotient update above
Also a lot of changes:
Bubble shapes for pending events are fixed.
You can set device name when logging in. This becomes important as libQuotient begins to upload one-time device key as part of E2EE implementation.
Markdown is parsed automatically by default, and works with replies.
Small UI improvements in timeline and room list.
Continuum
yuforia has continued work on Continuum, a desktop client written in Kotlin:
Continuum now preserves media content URI (mxc://) internally in order to treat them specially, instead of converting to all URI to http (or https) upon receiving.
This weekās version never considers cached mxc resources stale and no network request will be performed for refreshing.
Continuum also loads previews for http image links in text messages automatically. The usual http cache control rules are still followed in those cases.
Join #tkmc:matrix.org to chat more about Continuum, or about koma, the underlying library.
Riot Web
Riot v1.3.0-rc.1 is now ready for testing at https://riot.im/staging. This includes some last minute polish of reactions and edits, and also adds initial support for soft logout. This release will have reactions and message editing enabled via configuration on riot.im once it stabilises.
Self-hosted installs that wish to do the same would need to alter their config.json in similar fashion. This is because these features currently depend on unstable APIs, and we donāt want to move them out of labs and fully on by default until that is resolved.
Riot iOS
Reactions and edits:
Enabled by default (no more LABS setting)
Reactions with non-unicode keys
Original event in the edit history (need homeserver update)
Upgraded rooms are now autojoined when tapping on the upgraded banner
File upload from the room screen and from the share extension
Thanks to Aaronās frequent mention of synapse-janitor and other such cleanup methods, Iāve finally gotten inspired enough to give it a try.
The playbook now contains a new Synapse Maintenance documentation page and an easy/safe way to run synapse-janitor.
To give an example, using synapse-janitor and a full Postgres VACUUM yielded a 29% reduction in disk space used by Postgres on my personal homeserver (5.3GB -> 3.8GB).
Alexey Murz Korepov also reminded us about synapse-purge, which weāve mentioned here before - but is designed for a similar purpose.
the avhost/docker-matrix image has moved to a debian buster base image, which got us an upgrade from python 3.5 to python 3.7.3 and jemalloc1 to jemalloc2, which should improve the performance of synapse.
I had/stole the idea to create a bot which uses message edits to send frames of an ASCII-art animation. I indeed created the bot, which works to a degree, but is quickly punished by rate-limiting, which limits the effectiveness. Still itās quite fun, you can check out the code here.
Half-Shot ābridged #synapse:matrix.org to #matrix-synapse on freenode to help folks who might be experiencing issues with their homeserver and need a IRC based support channelā
Black Hat made a cool-looking thing: āIt basically shows all pictures in this room in a waterfall, with āinfinite scrollāā
Thatās all I know š
See you next week, and be sure to stop by #twim:matrix.org with your updates!
Dendrite continues marching forward! As more attention is turned towards our fairly lengthy PR list, contributors who have not done so already are reminded to merge Dendriteās master branch into their PRs, as converting the project to go modules caused a lot of conflicts. A tag has been added to each PR that needs forward merging, visible here.
Weāve also got plans now! Plans for how to properly ship this thing over the coming months so look out for that soon!
Synapse 1.1.0 released
This week we shipped v1.1.0, which provides an overhaul of docker configuration, more authentication options and improved db io. Itās worth noting that v1.1.0 is the first Synapse release to drop support for Python 2 (and Postgres 9.4), this paves the way for using Python 3 only functionality.
Weāve been working on supporting soft logout, more edits and reactions support, open tracing support not to mention a complete rewrite of the push server Sygnal. Weāll be rolling out new Sygnal gradually over the next week or two.
Finally, aided by dropping Python 2 support, weāve been putting in a bunch of work to improve Synapse in resource constrained environments. This will be a constant theme over the coming months.
Dept of SDKs and Frameworks š
python-matrixbot
Brian Ć appeared to tell us about python-matrixbot. This is a project that has existed for some time.
A Python module meant to act as a base class for a Matrix bot.
The MatrixBot class will connect to the Matrix server, start a listener on each joined room, and listen for room invites from other users. It also includes helper methods you can use to extend the functionality. It is built on the Matrix Python SDK which can be directly accessed via MatrixBot.client
koma got some improvements, based on whatās learned developing Continuum, which is a desktop client based on it.
Make api calls suspendable functions (which are like Kotlinās flavor of async). This way, the caller donāt need to worry about forgetting to call await or a coroutine being left unstarted.
Borrowing from functional programming, model the outcome of a call as a discriminated union, which can be either a success or a failure. The successful case is optimized with inline classes, an experimental feature in Kotlin 1.3, and wrapping is avoided.
Make MatrixError a subclass of HttpError, because the http status code can be handy
Just released version 1.2.1 of the Ruby SDK, fixing an error in the media download URL generation
Dept of Bridges š
matrix-appservice-bridge release 1.9.0
Half-Shot was seen to exist IRL this week, he also found time for a new release:
Today we have a new matrix-appservice-bridge release 1.9.0. The bigname feature this week is a new store for mapping matrix events to remote ones, so bridges can handle changes made to sent events like reactions/threading/edits/redactions :). The reason for this feature appearing suddenly will become clear very soon.. š
mx-puppet-bridge (inc slack, tox, discord)
Another week, which means more work on the mx-puppet-bridge ecosystem! A new > bridge has been added, mx-puppet-discord. Soru finally added license files > (Apache-2-0) and some readmes.
This is the new puppeting bridge! The idea is that, in the long run, this will > be run in conjunction with matrix-appservice-discord Half-Shot/matrix-appservice-discord), where mx-puppet-discord handles DM > puppeting and matrix-appservice-discord the remaining. For this, the message > parsing was split in a new repository, matrix-discord-parser. The idea is that, in the > future, when inviting a ghost on matrix-appservice-discord itāll initiate > conversation within mx-puppet-discord
basic text messages
handle files
handle edits, deletes
mx-puppet-discord does only DMs, for non-DMs please use matrix-appservice-discord
Weāve got a dedicated room for slack bridge development over at #matrix_appservice_slack:cadair.com, since itās picked up in terms of community PRs and general interest. Itās not currently being used as a support room, however.
They mention,
warning may contain ranting about the codebase
But that could be any room, so it seems ok to me.
WARNING: LATE ADDITION
Hi everyone! Myself and Cadair have been working hard on a new Slack bridge release, and we are finally ready to push out a release candidate for 0.3.
The headline features are:
Implement message deletion.
Add support for edits.
Add support for reactions.
Add support for threading (using replies).
Support displayname and avatar lookups for Slack bots.
Replace channel mentions with canonical aliases for bridged rooms.
Support for slack attachments (Thanks @umitalp for the initial groundwork and @Cadair for the cleanup)
After an internal release, we are working on improving the performance, especially for initial sync and for navigation between rooms.
Also we are fighting bugs.
Iāve invested a lot of money in making Pattle happen
on iOS: MacBook, Apple Developer Program, and an iPhone.
Pretty costly, so any donations will be greatly appreciated!
What to expect in the next release:
Fix timeline jump issues
Remove redundant state messages when a room is upgraded
Start work on chat details screen (members, change name, etc.)
to push things forward on Matrix URIs front, Quaternion master branch now supports matrix:user/userid, matrix:room/roomalias and matrix:roomid/roomid URIs. For example, Quotient/Quaternion room can be opened by a link matrix:room/quotient:matrix.org.
This will be so much easier to use! Also:
Quaternion has got a new contributor, Roland Pallai (https://github.com/rpallai), who added colouring of messages sent by the local user and support of drag-n-drop of text and images on Quaternion, along with general improvements on the timeline. Many thanks!
Windows builds of Quaternion (CI and future releases) come with Qt Keychain enabled, storing your access tokens in Windows secure storage.
Bumped the K8s optimized Docker image to 1.1.0, with the same dropping of Python 2 and Postgres 9.4 support as the official image.
NB: The upstream docker configuration changes do not affect the K8s-optimized image, no configuration change is necessary to upgrade from 1.0.0 to 1.1.0
As announced last week, with the release of synapse 1.1.0 the avhost/docker-matrix image switched to running synapse with jemalloc by default
Dept of Services š
modular.im starting to make Small instances available
modular.im are making the much-asked-for SMALL instances available. This service is rolling out starting with people who have previously enquired about availability, which I gather is a lot of people. Go sign up if youāre interested!
The wait is almost over ... We're now rolling out our trial of Small Hosted Homeservers for Matrix. Have you got your golden ticket yet? ššļø pic.twitter.com/iUkAIHW9MY
weāve been working on a v1 admin dashboard for managing your Synapse instances through Modular. This is now live on the site and provides a basic suite of functionality including:
Viewing users of your synapse homeserver(s)
Creating users
Deleting users
Resetting user passwords
Viewing user profile and server access / activity
Sending messages to all system users as the system alerts user
Information about the synapse instance versions
Final thoughts š
Ananace is ācontinuing the rewrite of the release tracker project. Working towards getting it to only store state in Matrix so it can be run in a read-only environment like a K8s deployment.ā
Right folks, this is our first post 1.0 release, which means that we have now
officially dropped support for Python 2 and Postgres
9.4.
This means that we can start making use of Python 3 specific features and you
should expect lots of associated performance wins over the coming months. See the upgrade notes for more.
Synapse 1.1.0 also contains a reworked approach to the Docker image, as well lots
of performance improvements with special focus on DB IO - expect more to come
in this area.
Special thanks to community member Alexander
Trost for rounding out our SAML support and also
to Daniel Hoffend for contributing the ability
to disable local password authentication.
As of v1.1.0, Synapse no longer supports Python 2, nor Postgres version 9.4.
See the upgrade notes for more details.
This release also deprecates the use of environment variables to configure the
docker image. See the docker README
for more details.
No changes since 1.1.0rc2.
Synapse 1.1.0rc2 (2019-07-03)
Bugfixes
Fix regression in 1.1rc1 where OPTIONS requests to the media repo would fail. (#5593)
Removed the SYNAPSE_SMTP_* docker container environment variables. Using these environment variables prevented the docker container from starting in Synapse v1.0, even though they didnāt actually allow any functionality anyway. (#5596)
Fix a number of āStarting txn from sentinel contextā warnings. (#5605)
As of v1.1.0, Synapse no longer supports Python 2, nor Postgres version 9.4.
See the upgrade notes for more details.
Features
Added possibility to disable local password authentication. Contributed by Daniel Hoffend. (#5092)
Add monthly active users to phonehome stats. (#5252)
Allow expired user to trigger renewal email sending manually. (#5363)
Statistics on forward extremities per room are now exposed via Prometheus. (#5384, #5458, #5461)
Add āno-daemonize option to run synapse in the foreground, per issue #4130. Contributed by Soham Gumaste. (#5412, #5587)
Fully support SAML2 authentication. Contributed by Alexander Trost - thank you! (#5422)
Allow server admins to define implementations of extra rules for allowing or denying incoming events. (#5440, #5474, #5477)
Add support for handling pagination APIs on client reader worker. (#5505, #5513, #5531)
Improve help and cmdline option names for āgenerate-config options. (#5512)
Allow configuration of the path used for ACME account keys. (#5516, #5521, #5522)
Add ādata-dir and āopen-private-ports options. (#5524)
Split public rooms directory auth config in two settings, in order to manage client auth independently from the federation part of it. Obsoletes the ārestrict_public_rooms_to_local_usersā configuration setting. If ārestrict_public_rooms_to_local_usersā is set in the config, Synapse will act as if both new options are enabled, i.e. require authentication through the client API and deny federation requests. (#5534)
The minimum TLS version used for outgoing federation requests can now be set with federation_client_minimum_tls_version. (#5550)
Optimise devices changed query to not pull unnecessary rows from the database, reducing database load. (#5559)
Add new metrics for number of forward extremities being persisted and number of state groups involved in resolution. (#5476)
Bugfixes
Fix bug processing incoming events over federation if call to /get_missing_events fails. (#5042)
Prevent more than one room upgrade happening simultaneously on the same room. (#5051)
Fix a bug where running synapse_port_db would cause the account validity feature to fail because it didnāt set the type of the email_sent column to boolean. (#5325)
Warn about disabling email-based password resets when a reset occurs, and remove warning when someone attempts a phone-based reset. (#5387)
Fix email notifications for unnamed rooms with multiple people. (#5388)
Fix exceptions in federation reader worker caused by attempting to renew attestations, which should only happen on master worker. (#5389)
Fix handling of failures fetching remote content to not log failures as exceptions. (#5390)
Fix a bug where deactivated users could receive renewal emails if the account validity feature is on. (#5394)
Fix missing invite state after exchanging 3PID invites over federaton. (#5464)
Fix intermittent exceptions on Apple hardware. Also fix bug that caused database activity times to be under-reported in log lines. (#5498)
Fix logging error when a tampered event is detected. (#5500)
Fix bug where clients could tight loop calling /sync for a period. (#5507)
Fix bug with jinja2 preventing Synapse from starting. Users who had this problem should now simply need to run pip install matrix-synapse. (#5514)
Fix a regression where homeservers on private IP addresses were incorrectly blacklisted. (#5523)
Fixed m.login.jwt using unregistered user_id and added pyjwt>=1.6.4 as jwt conditional dependencies. Contributed by Pau Rodriguez-Estivill. (#5555, #5586)
Fix a bug that would cause invited users to receive several emails for a single 3PID invite in case the inviter is rate limited. (#5576)
Updates to the Docker image
Add ability to change Docker containers timezone with the TZ variable. (#5383)
Fix āUnexpected entry in āfull_schemasāā log warning. (#5509)
Improve logging when generating config files. (#5510)
Refactor and clean up Config parser for maintainability. (#5511)
Make the config clearer in that email.template_dir is relative to the Synapseās root directory, not the synapse/ folder within it. (#5543)
Update v1.0.0 release changelog to include more information about changes to password resets. (#5545)
Remove non-functioning check_event_hash.py dev script. (#5548)
Synapse will now only allow TLS v1.2 connections when serving federation, if it terminates TLS. As Synapseās allowed ciphers were only able to be used in TLSv1.2 before, this does not change behaviour. (#5550)
Logging when running GC collection on generation 0 is now at the DEBUG level, not INFO. (#5557)
Reduce the amount of stuff we send in the docker context. (#5564)
Point the reverse links in the Purge History contrib scripts at the intended location. (#5570)
This article concerns matrix-nio, and asyncio. Weāll build a simple āecho botā, meaning a bot which replies to messages with the text it has just read. Note that this article does not cover E2EE with matrix-nio.
Instantiation and Login
First create a new venv, and install matrix-nio via pip. On the command line, run:
Terminal window
python3-mvenvenv
sourceenv/bin/activate
pipinstallmatrix-nio
Next, create a new Python file, and open it for editing. Weāll import everything we require for this tutorial:
from importlib import util
import asyncio
from nio import (AsyncClient, SyncResponse, RoomMessageText)
Weāre importing asyncio so we can use the AsyncClient class from matrix-nio.
Create a new instance of AsyncClient by passing the homeserver and username as arguments:
Of course, we are using an async client, and awaiting the response. Because of this, we must call the async_client.login() from an async method, like so:
To get updates from a Matrix homeserver to the client, the client makes a request to the /sync endpoint. In the matrix-nio AsyncClient, this is wrapped by the sync() method. We can get the latest updates:
sync_response = await async_client.sync(30000)
30000 means we will wait up to 30 seconds before returning. sync_response will now contain a Python object containing a mapping of the (JSON) response from the Matrix homeserver. Weāll inspect this response in the next section.
In fact, we expect there to be updates regularly, so letās create a very simple loop:
while (True):
sync_response = await async_client.sync(30000)
print(sync_response) # note that this could be LARGE!
# do some reading from sync_response
In this way, every time there is a response (i.e. new events) from the homeserver, they are made available in sync_response for processing, and we loop again.
Explore the sync response object
sync_responsecan contain multitudes, depending on the rooms this user is part of, or has been part of. sync_response.rooms.join contains updates for the rooms which the current user is ājoined toā (meaning, is a member of.)
Of these joined rooms, we are (perhaps!) most interested in the events on the timeline. These are stored in timeline.events, see below:
iflen(sync_response.rooms.join) > 0:
joins = sync_response.rooms.join
for room_id in joins:
for event in joins[room_id].timeline.events:
print(event)
Message events are a specific type of event which contain an Instant Messenger message. We can check the type before proceeding:
for event in joins[room_id].timeline.events:
ifisinstance(event, RoomMessageText):
print (event.body)
In these cases, where the event is a message to a room, the body field will contain the message text.
Isolate specific message event objects
Knowing that we can get the message text from an event, we can read it to determine a response. Letās make a new variable and have it store some string weāll check for:
response_string = "!replybot"
Now letās suppose weāre in our /sync loop, and just received an event. We can filter messages that are meant for our bot as follows:
iflen(sync_response.rooms.join) > 0:
joins = sync_response.rooms.join
for room_id in joins:
for event in joins[room_id].timeline.events:
ifhasattr(event, 'body') and event.body.startswith(response_string):
print(event)
Use room_send
To send messages, matrix-nio provides a room_send() method. There are three arguments:
the room_id
the message type, we will use ām.room.messageā
a JSON object representing the content of the message
Letās improve the example above, by sending back a message to echo the ones we isolated above:
joins = sync_response.rooms.join
for room_id in joins:
for event in joins[room_id].timeline.events:
ifhasattr(event, 'body') and event.body.startswith(response_string):
Now whenever the bot receives a message ā!replybot some messageā it will send back āsome messageā.
Use of /sync next_batch tokens
Finally, letās consider the importance of next_batch tokens. Whenever you receive a response from the /sync endpoint, the response will contain a ānext_batchā field, which you then pass on the next request to ensure you have the latest messages. matrix-nio keeps track of this automatically, so it doesnāt get repeated messages. However, when you stop the program and call the .sync() method again, how can you tell it where to start from? First letās get the latest next_batch token:
A few weeks ago there was some
discussion around the privacy
of typical Matrix configurations, particularly how Riotās default config uses
vector.im as an Identity Server (for discovering users on Matrix by their email
address or phone number) and scalar.vector.im as an Integration Manager (i.e.
the mechanism for adding hosted bots/bridges/widgets into rooms). This means
that Riot, even if using a custom homeserver and running from a custom Riot
deployment, will try to talk to *.vector.im (run by New Vector; the company formed by the core
Matrix team in 2017) for some operations unless an alternative IS or IM has
been specified in the config.
We havenāt done as good a job at explaining this as we could have, and this
blog post is a progress update on how weāre fixing that and improving other
privacy considerations in general.
Firstly, the reason Riot is configured like this is for the userās convenience:
in general, we believe most users just want to discover other people on Matrix
as easily as possible, and a logically-centralised server for looking up user
matrix IDs by email/phone number (called third party IDs, or 3PIDs) is the only
comprehensive way of doing so. Decentralising this data while protecting the
privacy of the 3PIDs and their matrix IDs is a Hard Problem which weāre unaware
of anyone having solved yet. Alternatively, you could run a local identity
server, but it will end up having to delegate to a centralised identity server
anyway for IDs it has no other way to know about. Similarly, providing a
default integration server that just works out of the box (rather than
mandating the user configures their own) is a matter of trying to keep Riotās
UX simple, especially when onboarding users, and especially given Riotās
reputation for complexity at the best of times.
That said, the discussion highlighted some areas for improvement.
Specifically:
When doing work on making Matrix GDPR
compliant
back in May 2018, we set up a single privacy
policy
for the services we run, and got users to agree to it by locking them out of
the matrix.org homeserver until they did. However, we missed that users not
on the matrix.org homeserver might still be using our Identity Service (IS)
& Integration Manager (IM) without accepting the privacy policy. Over the
last few weeks weāve been working on addressing this - it turns out that
itās a pain to fix, given the Identity Service doesnāt have the concept of
users, so tracking which users have agreed to the policy policy or not means
some fairly major changes. The current proposal is to change the Identity
Service to use a form of OpenID to authenticate users against their
homeserver in order to check theyāve accepted the ISās terms of use - see
MSC2140 for the gory
details.
Meanwhile, Riot is being updated to prompt the user to accept the IS & IM terms
of use (if different to the HSās), and thus make it crystal clear to the user
that they are using an IS & IM and that they have the option not to if desired - see https://github.com/vector-im/riot-web/issues/10167 and associated
issues.
This includes also explicitly prompting the user as to whether they want
3PIDs they provide at registration to be discoverable, as per
https://github.com/vector-im/riot-web/issues/10091.
Riot on iOS & Android gives the option of scanning your local addressbook to
discover which of your contacts are on Matrix. The wording explaining this
wasnāt clear enough on Android - which we promptly
fixed. Separately, the
contact details sent to the server are currently not obfuscated. This is
partially because we hadnāt got to it, and partially because obfuscating
them doesnāt actually help much with privacy, given an attacker can just
scan through possible obfuscated phone numbers and email addresses to
deobfuscate them. However, weāve been working through obfuscating the
contact details anyway by hashing as per
MSC2134, which has all
the details. Weāre also adding an explicit lookup warning in Riot/Web, as
per https://github.com/vector-im/riot-web/issues/10093.
There was a bug where Riot/Web was querying the Integration Manager every
time you opened a room, even if that room had no integrations (actually, it
did it 3 times in a row). This got fixed and
released in
Riot/Web 1.2.2 back on June 19th.
Matrix needs to authenticate whether events were actually sent by the server
that claimed to send them. We do this by having servers sign their events
when they create them, and publishing the public half of their signing keys
for anyone to query. However, this poses problems if you receive an event
which is signed by a server which isnāt currently online. To solve this, we
have the concept of trusted_key_servers (aka notary servers), which your
server can query to see if they know about the missing serverās keys. By
default, matrix.org is configured as Synapseās trusted notary, but you can
of course change this. If you choose an unreliable server as the notary
(e.g. by not setting one at all) then thereās a risk that you wonāt be able
to look up signing keys, and a splitbrain will result where your server
canāt receive certain events, but other servers in the room can. This can
then result in your server being unable to participate in the room entirely,
if itās missing key events in the roomās lifetime.
Our plan here is to get rid of notaries entirely by changing how event
signing works as per
MSC1228, but this is
going to take a while. Meanwhile weāre going to check Synapseās code to
ensure it doesnāt talk to the notary server unnecessarily. (E.g. it should
be caching the signing keys locally, and it should only use the notary
server if the remote server is down.)
When doing VoIP in Matrix, clients need to use a TURN server to discover
their network conditions and perform firewall traversal. The TURN server
should be specified by your homeserver (and each homeserver deployment
should ideally include a TURN server). However, for users who have not
configured a TURN server, Riot (on all 3 platforms) defaulted to use
Googleās public STUN service (stun.l.google.com). STUN is a subset of
TURN which provides firewall discovery, but not traffic relaying. This
slightly increased the chances of calls working for users without a proper
TURN server, but not by much - and rather than fall back to Google, weāve
decided to simply remove it from Riot (e.g.
https://github.com/matrix-org/matrix-ios-sdk/commit/24832a2b14fb72ae6f051d5aba40262d11eef65d).
This means that VoIP might get less reliable for users who were relying on
this fallback, but you really should be running your own TURN server anyway
if you want VoIP to work reliably on your homeserver.
As you can see, much of the work on improving these issues is still in full
swing, although some has already shipped. As should also be obvious, these
issues are categorically not malicious: Matrix (and Riot) literally exists to give users full control and autonomy over their communication, and privacy is a key part of that. These are avoidable issues which can and will be solved. Itās worth noting that we have to prioritise privacy issues alongside all the other development in Matrix however: thereās no point in having excellent privacy if there are other bugs stopping the platform from being usable.
Weāll do another blog post to confirm once most of the fixes here have landed -
meanwhile, hopefully this post provides some useful visibility on how weāre
going about improving things.
Matthew & Amandine talk to Ross Schulman; new Matrix.org Foundation Guardian!
Dept of Status of Matrix š”
Hottest GovTech Startup in Europe at The Europas tech awards
New Vector won Hottest GovTech Startup in Europe at The Europas tech awards last night for work on rolling out Matrix for France and elsewhere!
It was mainly judge-based, but public votes were used to filter.
We're super proud to have won the hottest Gov/Reg/Civic Tech startup at @TheEuropas for creating self-sovereign secure communications on top of @matrixdotorg for the Public Sector - particularly with @_DINSIC and @tchap_dinsic! (So proud we created a twitter account at last :D) pic.twitter.com/PGqoHx007T
MSC2134 Identity Hash Lookups is making steady progress
Readers who like crypto/hashing are encouraged to join in the fun
MSC2140 Terms of Service for ISes and IMs is nearing a conclusion
MSC2108 Sync over Server Sent Events hopes to add an alternative sync method to long-polling
Finally servers pushing events!
MSC2010 Adding client-side spoilers is a small feature but requires a lot of thought to integrate it seamlessly
Sorunome thought this would be easy
Lots of small fixes across the board.
Dept of Servers š¢
Synapse
Hereās Neil:
With 1.0 shipped we are now starting to take a closer look at Synapse performance more generally and this will be a theme for us over the coming months. We want to improve not only large scale deployments such as Matrix.org but also optimise for smaller instances.
You may have seen a few trial servers run by core team members in matrix.org community rooms popping up and this is a precursor for a broader effort to make synapse more manageable on less powerful infrastructure. My own instance has been sat at a pretty steady 256MB of RAM.
Other than that, based on 1.0 feedback, we have been working on improving the Synapse upgrade path and expect to put out a new release next week containing the tweaks. Specifically this means improving configuration for Docker installs, and configuration management for sending emails.
Weāre also implementing open tracing into Synapse, initially to help with e2ee debugging, but it will make tracking down strange behaviour easier more generally.
Finally weāre bringing our push server Sygnal kicking and screaming into 2019 and will upgrade to Python 3, drop gevent for twisted and update our vendor specific libraries, not mention improving the monitoring and alerting. Weāll also add in open tracing which will help hunt down push failures.
Finally finally, look out for a DAG visualisation tool written by GSOCer Eisha referenced elsewhere in TWIM - we consider this to be seriously cool, and canāt wait to start using it in anger.
This week was spent working on a big revamp of ruma-events, the library that defines Rust types for the āeventsā used in Matrix.
After some discussion in #ruma:matrix.org, I decided to make a move towards treating ruma-events as a higher-level library. Previously, ruma-events has more or less offered Rust types that are exact representations of the JSON structures used by Matrix. However, by representing events this way, it would be possible for users to easily create values that, while valid JSON, would be invalid events according to the specification.
The way weāre approaching this problem is by separating serialization/deserialization of JSON from validation of events.
Jeon 0.9.0 release. It is a release candidate for Jeon 1.0.0 which complies with the Matrix stable release 1.0.
Not a lot changes, just added missing endpoints and events.
Also I started to work on JeonServer, a Matrix server written on java.
Jeon is a set of Java interfaces to Matrix APIs, JeonServer is a proposed homeserver.
matrix-media-repo has received some speed improvements and is generally nicer to memory when using the s3 datastore. Give it a try and leave feedback in #media-repo:t2bot.io.
The GSoC project āMatrix Visualisationsā has made good progress during this first period:
The implementation of the CS API backend has been completed to properly retrieve events from a room in real time.
Many features have been added to the UI, here are some of them:
The DAG is displayed vertically, every node of the same ādepthā are on the same level in the graph and each node has outgoing arcs for each of its previous events (if they have already been retrieved).
The node at the top of the DAG allows to fetch earlier events by selecting it.
Each node can have two different colors whether its āoriginā is the HS the application is currently talking to or not.
The full JSON body of an event can be displayed by double clicking on its node.
It is possible to select which fields of the events will be directly included in the labels of the nodes.
A (server-side) backend has been implemented so that the application can directly talk to the PostgreSQL database of Synapse. You can find it on this repo.
Note that the support of the Federation API has been postponed so I could work on this Synapse database backend.
The UI of the application isnāt very beautiful or well-organised yet, as the effort is focused on the backends and core functionalities for now, but improvements will be made once these functionalities will be completed.
matrix-bot-sdk v0.4.0-beta.1 has been published with a bunch of improvements for appservices. Thereās still more planned before the final v0.4.0 release, however live testing is always better than unit tests. If you use the library, try npm install matrix-bot-sdk@0.4.0-beta.1 and report any issues to #matrix-bot-sdk:t2bot.io.
Just cut a 1.2.0 release of the Ruby SDK, including fixes for timeout handling, some general code cleanup and documentation work, and a collection of getters and setters for most of the specced room state types
Dept of Bridges š
freenode IRC bridge restarted
The freenode IRC bridge was restarted with the following fixes:
Some characters in IRC usernames (such as ā|ā) would cause IRC userās messages to not appear on the Matrix side.
This is work on the underlying bridge, protocol implementations still have to do their thing
room name, icon and topic (was already implemented but untested)
ghost name and icon (was already implemented, but untested)
file sending in both directions (image, audio, video, file, autodetect) (was already implemented, but untested)
have own ghosts create rooms ā> appservice bot is not needed, more invisible bridging (especially nice for 1:1 rooms!)
implemented 1:1 room bridging
implemented provisioning interface via appservice bot
remote -> matrix presence handling
remote -> matrix typing notifications
matrix -> remote puppet name tracking
matrix -> remote puppet avatar tracking
mx-puppet-bridge uses caching for better performance
Added linting
mx-puppet-slack
Lots of work here, smoothening things out
room name, icon and topic bridging
ghost name and icon bridging
file sending in both directions
presence and typing notifs
mx-puppet-tox
This is a new one! It bridges tox over to matrixā¦or, well, more acts like a client (as tox doesnāt have multidevice). Basic chatting was already functioning with only around 300 lines of code! The node toxcore bindings seem to only include support for 1:1 chats (and not the new group chats), so only that is implemented.
We are finalizing the MVP of RiotX. Many new features, along with many bug fixes this week:
Notifications for version with or without Firebase Cloud Messaging
Reply in e2e rooms
Change of DI tool (We are now using dagger2)
New settings, split into categories
New set of Emojis for quick reactions
New application icon
And many other little features
New disclaimer screen, displayed at first startup
New suggestion screen (based on bug report screen)
Min SDK version has been set to API 19 (Kitkat), mainly for security reasons, but also because we are using MotionLayout which is available only on API 18+.
Remaining work to do before we can release the first beta version on the PlayStore:
Our company, Scintillating, has integrated Riot as an end-to-end encrypted chat, video, and voice call provider for our decentralized scientific study management system Delphus. We have created a method of linking Matrix IDs with Ethereum addresses to allow scientific researchers to look up participants and create chat rooms to talk with individuals in a privacy-preserving manner.
Spectral
We didnāt get a Spectral update for a few weeks! Black Hat reported:
Quite a few changes in Spectral in the past few weeks. The room list filter is improved, and it only shows rooms with unread notifications by default. User can optionally hide join/leave events. Empty avatar in direct chats is fixed. Each user now has a unique message bubble background color in the timeline.
A new version of Pattle has been pushed to F-droid!
Although this isnāt the biggest release, itās still a big step: the first release of iOS will be available! The build is currently still in review by Apple.
You can download the iOS app via TestFlight soon, join #app:pattle.im to get the link immediately when itās available!
Other changes
Add .well-known support!
Automatically update homeserver property in āAdvancedā while typing username
Fix messages being wrongly grouped by sender
Fix direct chats staying marked as direct when someone else joins
If you would like to support me, you can now do so
via Liberapay and Patreon.
Iāve invested a lot of money in making Pattle happen on iOS: MacBook, Apple Developer Program, and an iPhone.
Pretty costly, so any donations will be greatly appreciated!
continuum tweaked the appearance of placeholder avatars. To make most users appear visually distinct, continuum has always used colors based on checksums and usernames to generate placeholders if any user doesnāt have an image as avatar. In previous versions, it always used two characters of the name. In the new version, if the username contains ideographic (usually east-asian) characters, a single character would be used. The reason is that the number of ideographic characters is vast and duplicates are less common, and most of them are close to a square or circle in shape so a single one would fit the GUI component better.
Last weekend hummlbach (from the UBports community) visited me and we worked 18 hours on implementing end2end encryption. We are now able to send encrypted messages. Key sharing and decrypting will follow.
Not available in the released version yet, but join #fluffychat:matrix.org for more info. Also: 18 hours! Woah.
Fractal released a 4.1 development version, which was added to the beta channel of flathub. danigm is eager to get 4.2 out soon and is trying to fix the last few bugs we want to see gone before then. He already opened a few merge requests.
riot-web
Updated the reactions UX with a new tooltip-style interaction
Added edit history dialogue and other small editing tweaks
Riot Desktop config options (such as default HS, theme, etc.) can now be overridden for easy customisation
riot-iOS
New reactions in the event contextual menu
Edits in e2e rooms
Still refining reactions and edits
matrix-enact, read-only client which performs the contents of rooms with Web Audio API
This is a read-only Matrix client, which takes the contents of a public room and āenactsā it, that is, it performs it using the Web Audio API in your browser
The original intention was to be a demonstration of what can be done with /context endpoint, but the project scope expanded a little. Hopefully people find it fun!
matrix-docker-ansible-deploy has seen a lot of work on bridging lately.
All currently existing bridges (Discord, Facebook, IRC, Telegram, Whatsapp) have been redone in a way that makes their configuration completely playbook-managed, as well as extensible.
Besides this, with Synapse v1.0 already out, weāve taken the opportunity to simplify the installation instructions a bit.
If you havenāt upgraded recently, now would be a great time. As always, be sure to take a look at the CHANGELOG before doing so.
I forked mxisd (https://github.com/ma1uta/mxisd) and will provide support this project. You can ask about help in a new room #ma1sd:ru-matrix.org
A new temporary name will be ma1sd (thanks Dandellion ).
Due to changing maintainers I start to prepare the new 2.0.0 release and should audit code and dependencies.
Also I forked matrix-synapse-rest-password-provider (https://github.com/ma1uta/matrix-synapse-rest-password-provider) because it often uses with mxisd.
Docker image, ansible support, debian, nixos and archlinux packages are temporary unavailable due to code auditing and changing maintainers.
Link existing unix accounts with accounts on a Synapse homeserver
Bot works by taking a list of emoji + responses from a user, then makes a new message event with those emojis each voted for once. In this way, you can quickly make a reaction-based poll.
The reactbot I announced last week has been updated to support arbitrary response content instead of just reactions. As examples, the repo now has a stallman interjection bot (ported from this and a replacement for jesaribot. Reactions as responses are still supported and the repo still has the TWIM cookie bot example too.
Final thoughts š
Ananace has been looking again at a release tracker theyād previously been working on.
kernel.org have set up an ActivityPub instance (see https://people.kernel.org/about.) Not strictly Matrix but interesting that they decided to move to a federated platform.
First GSOC evaluation submissions were due this week, all four Matrix projects are proceeding well. See Eishaās update above, and the other three last week.
red_sky, nheko maintainer, was seen to say: āI know there hasnāt been much activity from me lately. I was on vacation last week. Iāll be getting back to work on 0.7.0 todayā
The Europas awards ceremony was held at a venue next to my old flat. Small world! Small flat too.
Thatās all I know š
See you next week, and be sure to stop by #twim:matrix.org with your updates!
Matthew and Amandine talk about what weāre hoping to ship in June
Dept of Servers š¢
Dendrite
We have a whole lot of Dendrite news this week! First, from anoa, on the Matrix team:
Progress has been picking up this week with some of Cnlyās manyPRs finally getting merged! Cnly is our resident Google Summer of Code student this Summer and has been making strides ever since. We also finally fully migrated to BuildKite for Continuous Integration (which is much quicker than TravisCI) and have implemented golangci-lint as the projectās opinionated linter (catching some bugs in the process).
As part of CI, weāve set up Sytest for Dendrite, which grants us a loose method of keeping track of progress as we go along. Currently the only way to do so is to take a look at the testfile which holds all the tests we know Dendrite to be passing, and cross-referencing that with the list of all sytests (which isnāt actually available anywhere). Weāre hoping to make some nice way of visualizing progress over time with this data (and possibly break it down into Federation/Client/Application Service categories), but this will take some time.
Weād also like to thank some new contributors that have shown up since Matrix 1.0ās release. serra-allgood contributed a fix for querying aliases when using at least one application service, while DrGlitchMX sent a PR in for some database fixes. SUMUKHA-PKās PR to add room tags is still around and should see some love very shortly.
As far as the current goals of the project, weāre looking to primarily make Dendrite federate with Synapse, which will allow it to actually be usable for testing/basic day-to-day usage. We intend to have it done so in the next few months, so stay tuned!
Next, from Cnly, who is doing GSOCš on Dendrite:
For Dendrite, the first phase of GSoC has primarily seen improvements for the existing code including bug fixes, refinements for testing, and some refactoring that prepares Dendrite for later feature implementation. Among the changes, two important ones will be the initial support for EDUs in /sync responses as well as a federation destination cache that allows for more effective federation.
In the next phase, more focus will be put into feature completion, mainly for the Client/Server API component. Since this part of work covers various still-under-construction areas in Dendrite, it is foreseeable that more progress will actually be made than proposed.
Work is underway to bring Ruma up to date with version r0.5.0 of the Matrix specification. Starting with the most foundational libraries and working up towards the higher-level ruma-client, this work should be done in the next week or two. The bulk of the work since the last update has been on ruma-events, adding all of the events that were previously missing, and doing a full pass through existing events to make sure our definitions match the specification.
I canāt wait to see more updates out of Ruma!
Dept of Bridges š
matrix-appservice-discord 0.5.1
This announcement is nearly a week old from Half-Shot, coming in right after the previous TWIM was published:
Option to limit maximum document size when bridging so it wouldnāt download 1.5gb files and run out of ram
Made the state cache get updated when sending state events. Not doing this was causing some problems on t2bot.io, which has disabled echoing state events (probably for performance reasons)
Reliable Bridges GSOC š
Kai is working on bridging in Matrix, mentored by Half-Shot:
The first phase of GSoC is nearing its end and progress has been made on the Reliable Bridges project.
The main focus of the work was on delivering a running implementation for the signaling of permanent errors. Work has been done on matrix-appservice-bridge together with matrix-appservice-discord. An internal version of the Discord bridge with the new feature is already running and the related PR for the bridge SDK is in the state of being completed.
The next phase of GSOC comes with new goals. First, the preparation of a MSC for the signaling of bridge errors, so that the feature can be discussed in the community and can become a part of The Spec. The second goal will be the modification of Riot Web, so that the user is notified when an error condition occurred.
There also has been a slight change of plan: The work on notifying clients when parts of the system are temporarily unavailable needs reconsideration as not even the core Matrix network itself has this capability. The way forward here should probably be a unified solution.
Soru started working on a new puppeting library, for a lack of a better name it is currently called mx-puppet-bridge.
Unlike the existing puppet bridge, the focus of this one is to handle multiple users at once, all dynamically without needing to edit the configuration file over and over. Additionally the new codebase results in quite a few things smoothened out for easier protocol implementations.
Furthermore mx-puppet-bridge is written in typescript and based on travisā matrix-bot-sdk.
As protocol implementation for testing, soru also wrote mx-puppet-slack (again, for the lack of a better name). Basic message sending in both directions is already fully functional.
Exciting stuff! And all the more so for the third-person announcement style!
Added support for rendering redactions and edits (rendering reactions coming up next)
Added commands to create rooms, start private chats and edit room tags
Rendered images now donāt take up the whole width
Improved memory usage slightly
Broke everyones existing caches (rm -rf ~/.cache/gomuks should help if it panics at startup)
The command for creating rooms and the image resolution limit were added by a new contributor, J. R., who is currently working on adding .well-known discovery support.
Iāve been having a good play with this and like it at lot!
Improved support for more types of events in
continuum this week:
invitation events are supported
support for events that typically appear when a room is being set up is in progress
events that are not yet supported are displayed with a fallback view. Blank rows in previous versions are now fixed. You could always right-click on any event to view the source, of course
Pattle, Dart client
Wilko came in with this thorough update, Pattle development is MOTORING:
A new version of Pattle has been pushed to F-droid!
This release is mostly focused on bug fixing and bug reporting! This is why I urge all users who have been having problems before to try Pattle again! Chances are that your problem has been fixed, and if not, it will be reported with more information so I can fix it!
When an error is reported, this data is sent:
Operating system version
Device model, brand, manufacturer and whether itās a simulator
A unique ID based on your device
In some errors the homeserver domain is logged, I will try to prevent this in the future.
This release also includes preparation for an iOS release next week!
Fixes and other changes:
Handle rooms that the user has left (a notice is shown that you canāt send any messages)
Show a date header above the chat creation event (not the first known event in the list as before)
When an error occurs during sync, show a message, including the Exception name
Fix replies causing an error if the formattedBody does not adhere to the spec (thanks to Mathieu!)
Fix errors not showing when logging in
Fix loading spinner showing when checking username or logging in even if loading took less than 3 seconds
Fix direct chats not detected when adder after the initial sync
Some general syncing issues have been fixed (causing the dreaded infinite loading spinner).
To install this release, add the following repo in F-droid:
If you would like to support me, you can now do so via Liberapay and Patreon. I actually need money for the Apple Developer program, which costs 100 euros per year to release Pattle on the App Store and Testflight.
Dept of Encryption š
Quotient (former QMatrixClient)
aa13q is a GSOC š student, working on E2EE and mentored by Kitsune:
My PR to the avhost/docker-matrix image (formerly the silviof) adding jemalloc support has just been merged. The image will now run synapse using jemalloc by default, which has been shown to reduce memory usage.
The first image is tagged as avhost/docker-matrix:jemalloc if all goes well the next synapse release will include jemalloc by default
Dept of Services š
New (web) forum from Federated Networks Association
Feneas (short for Federated Networks Association) just launched a new discussion forum aimed at creating a collaboration space for the federated web folk. Of course there is a category for Matrix as well. Have a peek! More info here: https://feneas.org/federated-networks-forum/
Should you want to follow the Matrix.org blog using your Diaspora protocol compatible account, there is an unofficial mirror of the blog at matrixorg@social.gibberfish.org. (disclaimer: Iām not affiliated with the account, just found it and reporting)
Final thoughts š
š¤ A company named āSlackā had an IPO, in which the share price closed up 50%. Slack are involved in āInstant Messagingā software and their product is known for the ability to connect to other services.
ā° Itās the first GSOC assessment NEXT WEEK. Weāll get updates from all four projects, but notice that three of the projects have already provided updates above.
āļø Black Hat (from Spectral and more) is āexperimenting with ruma-client-api and plans to create something awesome with itā, Alexandre Franke (from Fractal) got a new laptop. Fox ājust did [their] last java uni exercise ever and am now free from this curse, will have time to work on Neoā.
šØ This website has had a lot of polish over the last couple of weeks. Weāll be focusing on the incremental improvements still needed, and also on adding new documentation. Let us know what you think we should do next.
Thatās all I know š
See you next week, and be sure to stop by #twim:matrix.org with your updates!
Matrix Live - Prof. Jon Crowcroft, Foundation Guardian š
Matthew & Amandine talk with Prof. Jon Crowcroft, one of the new Matrix.org Foundation Guardians!
Matrix 1.0
The big news this week is that we declared ourselves to be out of beta. Youāll want the full post for all details, but hereās a taste:
We are very excited to announce the first fully stable release of the Matrix protocol and specification across all APIs - as well as the Synapse 1.0 reference implementation which implements the full Matrix 1.0 API surface.
This means that after just over 5 years since the initial work on Matrix began, we are proud to have finally exited beta!! This is the conclusion of the work which we announced at FOSDEM 2019 when we cut the first stable release of the Server-Server API and began the Synapse 0.99 release series in anticipation of releasing a 1.0.
As part of the 1.0 we also announced the Matrix.org Foundation.
For the full update on the Foundation, please check out the new website content at https://matrix.org/foundation which should tell you everything you could possibly want to know about the Foundation, the Guardians, the Foundationās legal Articles of Association, and the day-to-day Rules which define the Open Governance process.
Thanks to everyone who has stuck with us along the way to make this possible.
Synapse
We released Synapse 1.0.0 this week to coincide with Matrix 1.0. As a release Synapse 1.0.0 focuses on security and stability which in turn builds a firm foundation for the performance improvements that you can expect to see over the Summer. Read all about it here.
Crypto
We have not one but two working end to end demos of device cross signing! This is going to be huge and is key step towards being able to make Matrix e2ee by default.
Cross Signing Demo
Additional Treats
But wait thereās more! Pantalaimon not only provides a way for clients and bots to participate in e2ee rooms, it now provides the ability to search. Checkout this (religious themed) demo.
Pattle
Pattle is going great guns at the moment. Hereās Wilko:-
A new version of Pattle has been pushed to F-droid!
Add ability to create group chats!
Show chat creation events (āWilko has created this groupā)!
Show emote messages correctly!
Handle display name changes! Display names of messages will now be as they were at time of sending.
Donāt show invite and join events in direct chats This is only happens for the two initial users in the direct chat. If someone invites someone else to the direct chat (trough another client), the invitation will show up in the timeline.
Use the timeout parameter while syncing. This means that receiving new messages should be way quicker! (Thanks Mathieu!)
Store messages retrieved remotely (thanks Mathieu!) This means that scrolling up in a chat will be faster now, because the messages are cached.
Always show a date header above the oldest event
Show replies correctly in chat overview
Show sent state icon next to own message in chat overview
Show newly joined rooms at the top in the chat overview
Use a bit bolder font for chat names in overview
To install this release, add the following repo in F-droid:
There is now also a room for the Matrix Dart SDK (which is being developed for Pattle): #sdk:pattle.im
If you would like to support me, you can now do so via Liberapay and Patreon.
Pattle
Pattle
RiotX (Android)
Crypto has been merged to develop \o/. We are still working on the feature, for the remaining actions: delete device, export and import keys, keys backup / SAS UI polishing, cleanup keys when signing out, and also fixing bugsā¦
A new screen has been added to create Rooms.
Animation of the Floating Action Button on the catchup screen has been improved.
Valere has started to work on notifications.
FranƧois is working on migration to Dagger2 (instead of Koin). It should improve performance and will allow us to implement multi-account support(!)
Copyright, Term and conditions, privacy policy and third party license screens are coming soon.
Progress indicator on Home for initial/catchup sync is coming soon as well.
Riot Android
Fix issue on SAS, to allow cross-signing.
Riot iOS
Work continues on edits and reactions - weāll have something for you very soon ā¦
Riot Web
Editing now supports editing unsent messages(!!), editing emotes, and lots of polish
Reactions now instantly cancel when you redact them
Redactions now instantly redact when you send them
Released v1.2.2-rc1 (up for testing at https://riot.im/staging/) - this release includes room breadcrumbs being out of labs, some fixes for inviting by email, fixes uploads in chrome canary / firefox nightly, config file validation, some fixes for the new emoji font and lots of other bug fixes.
New versions of the ruma-api and ruma-api-macros libraries were released, and work is underway to bring ruma-events up to date with client-server spec r0.5.0
I wrote a bot to control ansible playbooks https://github.com/Half-Shot/matrix-ansible-bot. Iām informed such a thing already exists as an ansible module, but I failed to realise that and wrote a independent one in TypeScript.
For room admins wondering if they can upgrade their rooms to v5, I made a bash script that checks all the servers in a room and prints a nice summary of the number of servers and members on each version: https://gist.github.com/tulir/aa2df287a0d192b86e5b675687791d16
Currently grin is building a backend for matrixservers.net that will collect data and push statistics every hour to our website. Next to that I wanted to push forward that me and several others are looking into building a Code of Conduct that should help every new and existing home server. This code of conduct will be build and curated from scratch in-order to help and support the network or any general project.
We are very excited to announce the first fully stable release of the Matrix protocol and specification across all APIs - as well as the Synapse 1.0 reference implementation which implements the full Matrix 1.0 API surface.
This means that after just over 5 years since the initial work on Matrix began, we are proud to have finally exited beta!! This is the conclusion of the work which we announced at FOSDEM 2019 when we cut the first stable release of the Server-Server API and began the Synapse 0.99 release series in anticipation of releasing a 1.0.
Now, before you get too excited, itās critical to understand that Matrix 1.0 is all about providing a stable, self-consistent, self-contained and secure version of the standard which anyone should be able to use to independently implement production-grade Matrix clients, servers, bots and bridges etc. It does not mean that all planned or possible features in Matrix are now specified and implemented, but that the most important core of the protocol is a well-defined stable platform for everyone to build on.
On the Synapse side, our focus has been exclusively on ensuring that Synapse correctly implements Matrix 1.0, to provide a stable and secure basis for participating in Matrix without risk of room corruption or other nastinesses. However, we have deliberately not focused on performance or features in the 1.0 release - so Iām afraid that synapseās RAM footprint will not have got significantly better, and your favourite long-awaited features (automatically defragmenting rooms with lots of forward extremities, configurable message retention, admin management web-interface etc) have not yet landed. In other words, this is the opposite of the Riot 1.0 release (where the entire app was redesigned and radically improved its performance and UX) - instead, we have adopted the mantra to make it work, make it work right, and then (finally) make it fast. You can read the full release notes here. Itās also worth looking at the full changelog through the Synapse 0.99 release series to see the massive amount of polishing thatās been going on here.
All this means that the main headline features which land in Matrix 1.0 are vitally important but relatively dry:
Using X.509 certificates to trust servers rather than perspective notaries, to simplify and improve server-side trust. This is a breaking change across Matrix, and weāve given the community several months now to ensure their homeservers run a valid TLS certificate. See MSC1711 for full details, and the 2 week warning we gave. As of ~9am UTC today, the matrix.org homeserver is running Synapse 1.0 and enforcing valid TLS certificates - the transition has begun (and so far we havenāt spotted any major breakage :). Thank you to everyone who got ready in advance!
Using .well-known URIs to discover servers, in case you canāt get a valid TLS certificate for your serverās domain.
Switching to room version 4 by default for creating new rooms. This fixes the most important defects that the core room algorithm has historically encountered, particularly:
ā¦and lots and lots and lots of bugfixes and spec omission fixes.
That said, there is a lot of really exciting stuff in flight right now which sadly didnāt stabilise in time for Matrix 1.0, but will be landing as fast as we can finalise it now that 1.0 is at last out the door. This includes:
Editable messages! (These are in Synapse 1.0 and Riot already, but still stabilising so not enabled by default)
Reactions! (Similarly these are in develop)
Threading!! (Weāve planted the seeds for this in the new āaggregationsā support which powers edits & reactions - but full thread support is still a bit further out).
Cross-signed verification for end-to-end encryption (This is on a branch, but due to land any day now). Weāve also held off merging E2E backups into the Matrix 1.0 spec until cross-signing lands, given it may change the backup behaviour a bit. Once this is done, we can seriously talk about turning on E2E by default everywhere.
Live-tracking of room statistics and state in Synapse! (This is in Synapse 1.0 already if you check out the new room_stats and room_state tables, but we need to provide a nice admin interface for it).
Support for smaller footprint homeservers by reducing memory usage and stopping them from joining overly complex rooms.
Then stuff which we havenāt yet started, but is now unlocked by the 1.0 release:
Fixing extremities build-up (and so massively improving performance)
Rewriting Communities. Groups/Communities deliberately didnāt land in Matrix 1.0 as the current implementation has issues we want to fix first. MSC1772 has the details.
Rewritten room directory using the new room stats/state tables to be super-speedy.
Just to give a quick taster of the shape of things to come, hereās RiotX/Android, the all-new Riot client for Android, showing off Edits & Reactions in the wildā¦
ā¦and hereās a screenshot of the final test jig for cross-signing devices in end-to-end encryption, so you will never have to manually verify new devices for a trusted user ever again! We demoed a very early version of this at FOSDEM, but this here is the testing harness for real deal, after several iterations of the spec and implementation to nail down the model. + means the device/userās cross-signing key is trusted, T means itās TOFU:
So, there you have it - welcome to Matrix 1.0, and we look forward to our backlog of feature work now landing!
Massive massive thanks to everyone who has stuck with the project over the years and helped support and grow Matrix - little did we think back in May 2014 that itād take us this long to exit beta, but hopefully youāll agree that itās been worth it :)
Talking of which, we were looking through the photos we took from the first ever session hacking on Matrix back in May 2014ā¦
ā¦suffice it to say that of the architectural options, we went with #3 in the endā¦
ā¦and that nowadays we actually know how power levels work, in excruciating and (hopefully) well-specified detail :)
There has been an absolutely enormous amount of work to pull Matrix 1.0 together - both on the spec side (thanks to the Spec Core Team for corralling proposals, and everyone whoās contributed proposals, and particularly to Travis for editing it all) and the implementation side (thanks to the whole Synapse team for the tedious task of cleaning up everything that was needed for 1.0). And of course, huge thanks go to everyone who has been helping test and debug the Synapse 1.0 release candidates, or just supporting the project to get to this point :)
The Matrix.org Foundation
Finally, as promised, alongside Matrix 1.0, we are very happy to announce the official launch of the finalised Matrix.org Foundation!
This has been a long-running project to ensure that Matrixās future is governed by a neutral non-profit custodian for the benefit of everyone in the Matrix ecosystem. We started the process nearly a year ago back with the initial proposal Towards Open Governance of Matrix.org, and then legally incorporated the Foundation in October, and published the final governance proposal in January.
As of today the Foundation is finalised and operational, and all the assets for Matrix.org have been transferred from New Vector (the startup we formed in 2017 to hire the core Matrix team). In fact you may already have seen Matrix.org Foundation notices popping up all over the Matrix codebase (as all of New Vectorās work on the public Matrix codebase for the foreseeable is being assigned to the Matrix.org Foundation).
Most importantly, weāre excited to introduce the Guardians of the Matrix.org Foundation. The Guardians are the legal directors of the non-profit Foundation, and are responsible for ensuring that the Foundation (and by extension the Spec Core Team) keeps on mission and neutrally protects the development of Matrix. Guardians are typically independent of the commercial Matrix ecosystem and may even not be members of todayās Matrix community, but are deeply aligned with the mission of the project. Guardians are selected to be respected and trusted by the wider community to uphold the guiding principles of the Foundation and keep the other Guardians honest.
We have started the Foundation with five Guardians - two being the original founders of the Matrix project (Matthew and Amandine) and three being entirely independent, thus ensuring the original Matrix team forms a minority which can be kept in check by the rest of the Guardians. The new Guardians are:
Prof. Jon Crowcroft - Marconi Professor of Communications Systems in the Computer Lab at the University of Cambridge and the Turing Institute. Jon is a pioneer in the field of decentralised communication, and a fellow of the Royal Society, the ACM, the British Computer Society, the Institution of Engineering and Technology, the Royal Academy of Engineering and the Institute of Electrical and Electronics Engineers.
Jon is a global expert in decentralisation and data privacy, and is excellently placed to help ensure Matrix stays true to its ideals.
Ross Schulman - Ross is a senior counsel and senior policy technologist at New Americaās Open Technology Institute, where he focuses on internet measurement, emerging technologies, surveillance, and decentralization. Prior to joining OTI, Ross worked for Google.
Ross brings a unique perspective as a tech- and decentralisation-savvy lawyer to the Foundation, as well as being one of the first non-developers in the Matrix community to run his own homeserver. Ross has been known to walk around Mozfest clutching a battery-powered Synapse in a box, promoting decentralised communication for all.
Dr. Jutta Steiner - As co-founder and CEO of Parity Technologies, Jutta is dedicated to building a better internet - Web 3.0 - where usersā privacy & control come first. Parity Technologies is a leader in the blockchain space ā known to many as the creator of one of the most popular Ethereum clients, it is also the creator of two ambitious new blockchain technologies, Polkadot and Substrate, that make it easier to experiment and innovate on scalability, encryption and governance.
Parity has been pioneering Matrix enterprise use since the moment they decided to rely on Matrix for their internal and external communication back in 2016, and now run their own high-volume deployment, with end-to-end encryption enabled by default. Jutta represents organisations who are professionally dependent on Matrix day-to-day, as well as bringing her unique experiences around decentralisation and ensuring that Web 3.0 will be a fair web for all.
Weād like to offer a very warm welcome to the new Guardians, and thank them profusely for giving up their time to join the Foundation and help ensure Matrix stays on course for the years to come.
For the full update on the Foundation, please check out the new website content at https://matrix.org/foundation which should tell you everything you could possibly want to know about the Foundation, the Guardians, the Foundationās legal Articles of Association, and the day-to-day Rules which define the Open Governance process.
And finallyā¦
Matrix 1.0 has been a bit of an epic to release, but puts us on a much stronger footing for the future.
However, itās very unlikely that weād have made it this far if most of the core dev team wasnāt able to work on Matrix as their day job. Right now we are actively looking for large-scale donations to the Matrix.org Foundation (and/or investment in New Vector) to ensure that the team can maintain as tight a focus on core Matrix work as possible, and to ensure the project realises its full potential. While Matrix is growing faster than ever, this perversely means we have more and more distractions - whether thatās keeping the Matrix.org server safe and operational, or handling support requests from the community, or helping new members of the ecosystem get up and running. If you would like Matrix to succeed, please get in touch if youād like to sponsor work, prioritise features, get support contracts, or otherwise support the project. Weāre particularly interested in sponsorship around decentralised reputation work (e.g. publishing a global room directory which users can filter based on their preferences).
Finally, huge thanks to everyone who has continued to support us through thick and thin on Patreon, Liberapay or other platforms. Every little helps here, both in terms of practically keeping the lights on, and also inspiring larger donations & financial support.
So: thank you once again for flying Matrix. We hope you enjoy 1.0, and we look forward to everything else landing on the horizon!
