t2bot.io - a public integrations network for Matrix - has passed 1 Million known rooms and 8.3 Million bridged users. 10,000 of these rooms are attributed to Voyager (a bot which actively goes out to find rooms to map Matrix, with the map currently down for maintenance), leaving the remaining ones either bridged, previously bridged, or using a different integration offered by t2bot.io for free.
The 8.3 Million users are mainly Discord and Telegram users which have been brought over to Matrix through bridges. The stats say āexcluding Twitter-bridgedā because thereās 424,832 old accounts from back in the day when t2bot.io had a free Twitter bridge available. To further break this down, about 6.8M are Telegram users (12% of Telegram) and 1.3M are Discord (<1% of Discord).
For perspective, t2bot.io has about 569 Million events stored in its database and sees approximately 30 thousand people bridged daily from the wider world into Matrix through its bridges.
This post is just a milestone update, but it also serves as a reminder that running your own server/bridges is also possible. In fact, itās even recommended to have better control over your own data and avoid latency issues that large providers, like t2bot.io, can unintentionally introduce. Synapse is relatively easy to set up with minimal sysadmin knowledge (guide), and thereās always paid offerings like Element Matrix Services (for home and also for fun) and Beeper for a richer bridging experience than t2bot.io can feasibly provide.
It is to be noted that eQualitie has set-up public Matrix and Element instances for Ukrainian people who are struggling right now. If you are in Ukraine or are in touch with Ukrainian people who need secure communications, you can show them https://kyiv.dcomm.net.ua, https://odessa.dcomm.net.ua, or https://kharkiv.dcomm.net.ua depending on their location so they get instructions on how to create an account on Matrix and stay in touch
MSC3575 Sliding Sync work is progressing, and a number of new features have been added to the MSC with implementations in the built-in web client in the proxy. A list of changes include:
spec/proxy: the ability to filter the room list by room name.
spec/proxy: Aninitial flag on the room to distinguish between updates and the initial sync for a given room. This can be used as a hint to clients to determine if they should update/replace their local data for that room.
proxy: more efficient algorithm for determining overlapping sets of ranges, resulting in fewer bytes over the wire when scrolling.
proxy: the client implementation now supports sending basic text messages as well as issuing/join,/invite,/leave commands.
proxy: the client implementation now has a developer HUD which tracks the Tx/Rx bytes as well as a visual representation of the sliding window.
The net result is a basic but incredibly low bandwidth syncing client: 60.79KB to download an entire initial sync on matrix.org. Further improvements will be done on the client to make sure it doesnāt scale with the number of rooms on the userās account (it currently does because it naively adds a placeholder for each room in the list) to ensure it remains extremely snappy and a vision for what Sliding Sync can do in practice, right now. Please continue giving feedback on MSC3575 or in #sliding-sync:matrix.org as the API is still in development and will change depending on what clients require.
Hereās your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.
Heads up! You may have noticed some changes to the matrix-org/matrix-doc github repository this week. The first major one being that it no longer exists!
We have separated it out into two repositories:
matrix-org/matrix-spec - the source text of the spec. Issues and pull requests filed here should relate to the source of the spec.
matrix-org/matrix-spec-proposals - all Matrix Spec Change proposals. New MSCs should be filed against this repository as pull requests. Issues should not be filed against this repository (and no, the issue tracker cannot be disabled, as it would cause transferred issue link redirects to 404).
The primary motivation for this was to separate concerns of interacting with matrix-org/matrix-doc. Those interacting with the specās source would have to wade through long-lived spec proposals, while those only interested in spec proposals would need to filter out spec source files and other issues and PRs related to them. This also helps writing tooling against the repo either as it no longer needs to filter MSCs via issue labels.
Existing links to matrix-doc should be automatically redirected by github to the right place. MSC authors and developers should update their git remote settings to point to the new repos.
What happened in detail to achieve this was:
matrix-doc was renamed tomatrix-spec-proposals
matrix-spec was created
All files other than the āproposalsā folder were transferred frommatrix-spec-proposals tomatrix-spec
All open issues frommatrix-spec-proposals were transferred tomatrix-spec
This MSC asks for the simple change of adding a ādestinationā field to the Authorization header of federation requests. This would have the benefit of allowing reverse proxies - potentially handling traffic for multiple homeservers - to know where requests should be sent. Or for forward proxies, which could validate outgoing requests from a homeserver before theyāre sent.
There is currently some discussion on the fundamental idea of the MSCs, which will need to be resolved first before it can move forwards.
Quite a varied bag this week! Firstly we cut a release candidate for Synapse 1.54.0, which we hope to release next week. As ever, testing and feedback is much appreciated.
Our switch to Poetry for dependency management has come a few steps closer, by removing tox from our Continuous Integration, and converting some of our ad-hoc scripts to proper Python entry points.
Meanwhile weāve been continuing to knock out blockers to improving room join speeds, including better exception handling and logging, and some exciting work to reduce wasted work when a client disconnects while weāre still processing a request.
Weāve also been taking a wider look at performance problems, such as slow /sync and /login times.
I have been asked by numerous people over months if there are any kind of graphs comparing Synapse and Dendrite. Up until now, my answer has been āas far as I know, noā. However, this got me thinking whether I can use Complement to engineer some kind of performance testing infrastructure. After hacking on it for a day or two, I can now share some of the work Iāve done in this area. There now exists two new binaries in the Complement repo: perftest and perfgraph. The former will run some tests and output a .json file. The latter will consume these files to generate .svg graphs. The tests performed include:
Creating a bunch of rooms with different users. (currently 10 users and 50 rooms)
Joining a bunch of rooms with different users.
Sending messages into rooms with different users. (currently 100 messages)
Doing an initial sync for all users.
Updating the display name of all users.
Doing an incremental sync for all users.
The metrics are collected using docker stats. This means the entire container is measured, not just the homeserver binary (e.g any daemons or assistive processes are included in the metrics) and includes:
CPU time consumed
Memory consumed
Bytes transmitted/received
Disk read/writes
Time taken
Here are some results:
This was a surprising find and probably indicates that Dendrite has a memory leak somewhere when creating events. However, the fun doesnāt stop there. We can also use this tool to compare different versions of the same homeserver:
This allows us to quantify any performance improvements weāve been working on, and you can clearly see how weāre being more efficient with CPU in newer versions. But the fun doesnāt stop there either. We can also compare different flavours of the same version of a homeserver:
You can quantify the cost of running sqlite or postgres for your particular installation.
Care must be taken when interpreting results. Lower doesnāt always mean better. For example, a low readout on ācreate_usersā may be a cause for concern as it might indicate that the server is not hashing passwords correctly (e.g using bcrypt/scrypt which are designed to consume CPU/memory). Servers which donāt implement all of the specification will also be abnormally low on CPU/memory (e.g not having to calculate unread counts or handle push notifications naturally means less work to do so less resources consumed). Furthermore, these tests do not yet test federation, so expensive remote joins are not measured (remote joins require the joining server to check signatures, hashes, etc of a lot of events which doesnāt happen for local servers). That being said, I have hopefully illustrated how useful this tool can be for both server developers trying to improve their software and server admins who want to use the right server for their hardware. The graphs are still a work-in-progress and thereās a lot more that can be done in this area beyond a few days work, but itās a start.
Today weāve released Dendrite 0.6.5 which contains early push notification support as well as a number of fixes and improvements. This release includes the following changes:
Early support for push notifications has been added, with support for push rules, pushers, HTTP push gateways and the /notifications endpoint (contributions by danpe, PiotrKozimor and tommie)
Spaces Summary (MSC2946) is now correctly supported (when msc2946 is enabled in the config)
All media API endpoints are now available under the /v3 namespace
Profile updates (display name and avatar) are now sent asynchronously so they shouldnāt block the client for a very long time
State resolution v2 has been optimised further to considerably reduce the number of memory allocations
State resolution v2 will no longer duplicate events unnecessarily when calculating the auth difference
The create-account tool now has a -reset-password option for resetting the passwords of existing accounts
The /sync endpoint now calculates device list changes much more quickly with less RAM used
The /messages endpoint now lazy-loads members correctly
Read receipts now work correctly by correcting bugs in the stream positions and receipt coalescing
Topological sorting of state and join responses has been corrected, which should help to reduce the number of auth problems when joining new federated rooms
Media thumbnails should now work properly after having unnecessarily strict rate limiting removed
The roomserver no longer holds transactions for as long when processing input events
Uploading device keys and cross-signing keys will now correctly no-op if there were no changes
Parameters are now remembered correctly during registration
Devices can now only be deleted within the appropriate UIA flow
The /context endpoint now returns 404 instead of 500 if the event was not found
SQLite mode will no longer leak memory as a result of not closing prepared statements
Spec compliance, as measured by Sytest, currently sits at:
Client-server APIs: 76%, up from 65% last time
Server-server APIs: 95%, up from 94% last time
As always, you can join us in #dendrite:matrix.org for Dendrite discussion and announcements.
Hey folks, Iām back from holidays and Iām proud to say that hookshot is kicking out release after release. This release is especially large and contains a bounty of new features and fixes. Of note are:
JIRA Datacenter (On Premise) is now supported.
You can now configure fine grained permissions for users on the bridge by userId, homeserver domain or room membership (think spaces).
Generic webhooks has sprouted a versioned API, with v2 allowing for finer control over the output.
GitHub connections now include the closing comment when issues and PRs are closed.
GitHub connections will also notify a room when an existing issue has been relabled to one filtered by that room.
Figma now uses MSC3440 for comment threads.
But seriously go check out the release, there is way more there than I can include in this TWIM post. Happy webhookin, matrix gang!
In case you did not know, Hookshot can be installed via spantaleevās Ansible playbook. In addition to updates for this new release, the role has recently gotten some improvements and fixes by the community, so in case you had issues before, now is the time to try again!
Allows anyone to run github commands in rooms. User part of the room listed can actually go and create connections to anywhere (we use it internally at $work), and then there is a final random user who gets those permissions because they refuse to join that room :p
Two bridge updates from matrix.org in one week? You would be forgiven for thinking it was FOSDEM Christmas.
The highlights are:
Support splitting users from different homeservers into different IPv6 blocks. We used this at FOSDEM to split attendeeds into a different ILINE.
Added a new metric clientpool_by_homeserver which lists the states of IRC clients, by the top 25 homeservers. Useful to know who is using your bridge.
Add support for subscribing to moderation policies. This allows a bridge admin to subscribe to Matrix āban listsā and filter who can use their bridge.
This release also now requires Node 14.x as Node 12.x will be out of support very soon.
This week, weāve fixed a few annoying issues in NeoChat:
On platforms with a system tray, NeoChat no quits correctly when the system tray integration is disabled
Similar state events are now aggregated into a single message (so no more scrolling through āModeration has changed the access control listsā š)
Sending messages starting with a ā:ā works correctly again
Weāve also improved NeoChatās startup time from cache by a factor of several hundred, depending on the situation
Released v1.10.5 and also v1.10.6 as a hotfix for a crash
Surveyed users and collated feedback from search result ordering for improvements to the new search experience (enable it in Settings -> Labs in production)
In labs (you can enable labs in settings on develop.element.io or on Nightly):
Pin drop location sharing has been added
Fixed message ordering issues in threads
Fixed threads discovery when scrolling back in a room
Added a couple of new ways to access pinned messages, in the room header and info panel
Improved the reliability of pinned messages with edits
Making progress on the registration flow by setting name and avatar
In development:
We will be trying out Spaces on iOS at 16:00 UTC / 17:00 CET on Tuesday, 8th of March. Head over to #element-community-testing:matrix.org to hear the latest on all testing sessions!
Live location sharing has started
In labs:
An option to use the last member avatar and name in the timeline
a very simple invite-accepting bot as part of matrix-streamchat to more easily facilitate peeking into rooms over federation.
You run one on your guest-access enabled homeserver, invite it to your remote room, and now all guest accounts from your server know they are in fact allowed to publicly read the room over federation without joining themselves. https://git.pixie.town/f0x/matrix-streamchat/src/branch/main/autojoin-bot
Work on having a setup page for the bot was added which allows to easily add the bot to your room (sets up permissions and invites the bot as required)
Offers high quality video calls with around 6 or 7 participants as a small & simple, standalone app that allows anyone to drop into a video conference easily.
This is the first implementation of MSC3401: a spec for secure, federated voice and video conferences.
Integrations into other clients are coming soon, Hydrogen is working on it right now
Weāll also be developing a server to mix media streams so calls can scale up to much larger number of participants. The server will only ever see your encrypted media, so calls will stay secure and confidential.
A set of Rust library crates for working with the Matrix protocol. Rumaās approach to Matrix emphasizes correctness, security, stability and performance.
Weāre back (to TWIM)! Hereās some highlights from the past few months:
Stronger types for some things:
Base64 which ensures you always decode / encode with the appropriate base64 dictionary [@jplatte]
TransactionId which is simply a little more explicit than String and easier to generate a random instance of [@jplatte]
Stable support for lots of previously-unstable functionality that got stabilized upstream as part of the Matrix specification releases v1.1 and v1.2 [@zecakeh]
More fine-grained feature flags for MSC implementations (before: everything behind unstable-pre-spec, after: most things behind dedicated unstable-mscXXXX features) [@zecakeh]
All of these changes are available in version 0.5.0 which was released about two weeks ago.
Additionally, @zecakeh recently started implementing extensible events and doing some pretty heavy internal refactoring that will make our release process faster and hopefully make contributing to Ruma easier as well.
With all these changes weāre very close to supporting all of Matrix v1.1 (and Matrix v1.2 too!), with the only major omission being Secure Secret Storage and Sharing (SSSS). Some work on that was done as far back as last yearās GSoC but it was blocked on a medium-sized refactoring of how we handle account data. Iām planning to get that over the finish line over the next weeks.
Iām working on a little hobby project, The Matrix Scribe.
The Matrix Scribe helps us re-post or transcribe messages into matrix that we received from somewhere else, posing as different ghost users to represent the original authors.
Scribe will post this:
Ann Iām Ann.
Bob Hello! Iām Bob.
when I write this to @scribe-bot:
@julian /scribe as Ann
@julian Iām Ann.
@julian /scribe as Bob
@julian Hello! Iām Bob.
That is an illustration of the goal. The output part works already; the bot part isnāt implemented yet.
Why?
I want to encourage people to experiment and play with extending Matrix in creative ways.
As a building block for myself towards building more usual matrix bridges. Scribe is the input half of the matrix half of a bridge, so literally one quarter of a typical matrix bridge.
I hope that I myself and maybe some other people will find Scribe fun or useful in itself.
Iāll be speaking at the new HYTRADBOI conference (29 Apr) about building collaborative, open software on Matrix. All of the talks sound super interesting, highly recommended! š The talks will be public after the conference as well.
Dept of Ping š
Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.
Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.
Rank
Hostname
Median MS
1
cry.is
288
2
nognu.de
319
3
rustybever.be
383.5
4
conduit.grich.sk
465
5
bogus.ee
568
6
matrix.sum7.eu
650.5
7
matrix.awesomesheep48.me
653
8
dendrite.matrix.org
1544
9
xethos.net
1756
Thatās all I know š
See you next week, and be sure to stop by #twim:matrix.org with your updates!
The Foundation needs you
The Matrix.org Foundation is a non-profit and only relies on donations to operate. Its core mission is to maintain the Matrix Specification, but it does much more than that.
It maintains the matrix.org homeserver and hosts several bridges for free. It fights for our collective rights to digital privacy and dignity.
