Time Flies
It's been 10 weeks already! This year's Google Summer of Code is coming to an
end. I really enjoyed working on this project, which turned out to be very
varied in what parts of Ruma I worked with: I've added endpoints, modified and
added new event types, created new identifier types and worked on macros.
This year for GSoC I had a project of extending Ruma's API coverage. More
specifically, I would complete Ruma's coverage of Matrix's Identity Service API.
On this front, I was successful, and now Ruma has 100% coverage of those
endpoints! I also had plans to implement new MSCs, namely MSC1946: Secure
Secret Storage and Sharing and MSC2785: Event Notification Attributes and
Actions. Along the way, I would also work on resolving a few issues here and
there, which helped get myself better acquainted with Ruma's codebase.
Working on Ruma was a good learning experience for me. I had an opportunity to
do actual work on a project which is used in the real world. All the while, I
was learning and practicing many skills. Working with git, coding against a
formal specification (and dealing with the problems caused by this). Although I
did learn a lot at University, this was a very useful hands-on experience. And
as a bonus, I got to program in Rust for the whole Summer! Rust is a language I
particularly enjoy using, but which I unfortunately rarely have the opportunity
to use at school.
The Final Parts of the Project
The end of my project did not turn out to be as smooth as planned. As mentioned
in the previous blog post. I had originally planned to work on adding support
for Secure Secret Storage and Sharing, introduced by MSC1946. Although the
sharing part was possible to implement with the current macros available in
Ruma, storage requires a major refactoring of these macros. This means that
although I was able to do the groundwork for supporting MSC1946, secret
storage will not be usable until those macros are reworked.
The reason for all this trouble is that MSC1946 introduces an event
m.secret_storage.profile.[key_ID] where [key_ID] is a placeholder for the
ID of a key to be stored. This means there could be any number of event types
possible, instead of the until now predefined set of event types. I thus opened
an issue for this to be supported in Ruma.
Although secret storage will need to wait to be merged, secret sharing has
already been merged. This part of MSC1946, as its name suggests, allows
clients to share secrets between devices. As such, keys can be exchanged between
devices using the part of the Matrix Spec introduced by this MSC. In fact,
matrix-rust-sdk has already implemented the gossiping of cross signing
keys, building on top of Ruma's implementation of Secret Sharing.
As it wasn't realistic to include the major macro refactor in my GSoC project, I
decided to move onto MSC2785: Event notification attributes and actions. I
had kept this as an extra goal for my project, and seemed to be a good fit for
the situation. Unluckily though, when I read more into the MSC's text, I
realized I would encounter the exact same problem as I had with Secret Storage.
This MSC introduced the event type m.notifications_profile.<profile> where
<profile> is the name of the profile to store. I still powered ahead,
implementing the different endpoints and events, temporarily setting a fixed
event type name for m.notifications_profile.<profile>. This allowed the
grunt of the work to be done, meaning it would be ready for when Ruma gets
support for variable event names. Although the MSC isn't finalized as of yet,
all the currently laid out endpoints and events have been implemented in a
branch ready to be merged once possible.
Thus there are a couple of pull requests which are blocked and that will be
merged in the future, once variable event type names are supported:
Finally, I also worked on this issue:
To solve this last issue, I decided to learn how to write procedural macros. In
the first half of GSoC, I had created a new macro_rules! macro, largely based
on a pre-existing one. As I had very little experience in writing macros, and
zero experience in writing procedural ones, I decided to read a short series of
blog posts to get myself up to speed. Once that was done, I got to work: I
added the generation of From trait implementations inside the pre-existing
proc-macro code which generate the event enums. I also added a derive
macro for enums created manually. I really enjoyed working on this, which was my
first foray into Rust's procedural macros.
Recap of PRs for my GSoC
Start of GSoC with the Identity Service API
Middle of GSoc, implementing MSCs and adding types
End of GSoC
What a fun summer! I was hoping I would become more a part of the Rust community this year, and
I think I have. I learned how to open an MCP (Major Change Proposal), which is a process that
language and implementation changes not big enough for an RFC go through. I am involved in an
effort to create a new plugin lint system similar to Clippy. I made a few more contributions
to Clippy, fixing old lints and adding a new one. I am happy to become more involved in Ruma and
Rust and plan to finish all my works in progress.
Since I am working with more people than just my mentor, making progress seems to take longer.
I am glad for the experience working with multiple people, learning how to follow directions from
multiple people. You have to be able to write code in the "voice" of someone else or at least
meet them halfway. I know that this is an invaluable skill that will serve me well in my career.
Thank you to @jplatte and @iinuwa for all the help and encouragement!
Starting on Automation
Second Half
Progress on Automated Checks for Ruma
I think the work on the linter will be extremely valuable if we can get it working. The ability
to write crate-specific lints will be a huge help to library writers and users. I can imagine many such
lints for a complex library like Ruma. I am excited to be involved!
Hello all! This year, Ruma has the pleasure of having two students for Google
Summer of Code (GSoC) 2021: Devin (who worked on Ruma's proc-macro code during last year's GSoC) and Adam. We'll let them [re]introduce
themselves.
Ruma API Coverage GSoC
by Adam Blanchet
This year for GSoC, Ruma has two students! I am the "newer" one of the two, and
my name is Adam Blanchet. I've been at work adding support for more of the
Matrix Spec in the past few weeks, and will be continuing on doing so for the
remainder of GSoC.
Identity Service API
Although I had made progress before the official coding period for GSoC had
started, I continued with finishing the remaining unimplemented Identity Service
API endpoints:
Apart from the 3PID unbind endpoint which is currently blocked,
this concludes the full coverage of the Identity Service API. This means that
now projects built upon Ruma will be able to more easily integrate functionality
with identity servers. This could also enable easier development of an identity
server written in Rust!
Implementing MSCs and adding types
I then moved on to a few other miscellaneous issues, to get accustomed to
working on the other parts of the Ruma project:
-
Add client secret and session identifier types:
I found this PR to be more interesting than its title may indicate. Not only
did I add two identifier types, but I added a new macro to create validated
identifier types. The new macro was also used to greatly simplify the
declaration of ServerName struct. I really enjoyed working on this PR. It
was an occasion to create a new macro to simplify the creation of these and
new types. I had never written macro code before, and it was a good occasion
for me to learn something new!
-
Add RoomName struct:
This one did not involve macro code, though similar in principle to the
previous PR. It adds a RoomName struct to be used in an event, which
enforces a maximum length.
-
Update endpoints for Blurhash implementation:
MSC2488 adds the use of Blurhashes to the Matrix Spec. Although
some Blurhash support had been previously added to Ruma, I
extended this to all endpoints and events specified in the MSC.
-
Add "knock" feature from MSC2403:
This adds support for knocking on rooms, a feature which I'm looking forward
to see implemented in clients. It was introduced in MSC2403.
-
Implement reasons for leaving a room:
Adding this feature, introduced in MSC1983, seemed rather straightforward.
However, I bumped into an issue with a conditionally compiled field with a
lifetime annotation. Thankfully, @DevinR528 helped diagnose the problem
which was some macro code, for which they think they have found a
solution along with @jplatte.
Once that fix is implemented, we will also be able to merge my PR.
What's next?
I'm looking forward to implementing the MSC for Secure Secret Storage and
Sharing. This is the second large part of my project, it will involve
adding a number of events to support storing and sharing both keys and secrets.
To achieve full support, I would also need to implement crypto routines. I
expect that adding these will require more effort than adding the events, though
I look forward to the challenge.
I found my first half of GSoC to be very enriching. I really enjoyed working on
this project: I got to learn how to read and apply a specification, and I
learned the rigor required for a project such as Ruma. I have sometimes been
caught off-guard: sometimes I miss extraneous newlines, sometimes (often) I
forget to add a changelog entry, and other similar more stylistic rather than
functional issues. Ruma is a big project with many crates, yet it maintains
consistency throughout, a quality I greatly appreciate. I now know that this
consistency comes from the thoroughness with which each change is checked, and I
hope to improve my own rigor when it comes to each of my pull requests.
by Devin Ragotzy
It's been a busy few weeks! I have started the process of checking four todo
items off of Ruma's automated checks issue by contributing
lints to the Clippy project.
I temporarily broke Clippy, Oh No! Luckily, the
fix seems to have worked.
I've spent a lot of time familiarizing myself with the rustc usefulness
checker. The is how the Rust compiler determines if you have a match for all
possible patterns and checks if any of the patterns are useless.
let x = true;
match x { // `x` is matched exhaustively (all match patterns are useful)
true => {},
false => {},
// true => {}, would be un-useful
}
This is where I now plan to add a lint to check for missed patterns on structs
and enums that are marked non_exhaustive (see issue).
This is helpful to Ruma because all of our public types are marked in this way,
allowing us to update our types in a non-breaking way while still informing
users of the change. Also, for anyone interested, I saw an interesting blog
post about usefulness checking.
For the last few days, I have been working with my mentor @jplatte to find
and fix a very sneaky issue with conditionally compiled fields and lifetime
annotations. This was found by my fellow GSoC participant in this
PR. I think we have finally found a solution, fingers crossed.
I have thoroughly enjoyed the first half of this year's GSoC! I'm excited to
contribute to back the compiler for the language we all love so much ;). In the
second half of GSoC I hope to get ruma-check to a
usable state adding the few checks left from Ruma's automated
checks.
Devin and Adam have already made it through the first half of GSoC and have made
good progress. We're looking forward to the rest of the summer with these two!
Wow the summer has flown by, it feels like just yesterday I was learning how to
rebase and what exactly it is Ruma does. I exaggerate slightly, but it is a big
library with lots of public API surface. I have learned more in the last few
months than in two years of school. I have been able to observe and participate
in a project with a community growing around it, been a part of discussions
about design and best practices, given and received numerous code reviews as
well as learned the process of addressing the feedback, and working from a
specification. In short, this has been an amazing opportunity to gain
experience in all the things that are hard to obtain in a classroom.
My project goal was to improve the existing macros in ruma-events-macros and
ruma-api-macros. It became clear early on that this would include some major
API changes and that improving the macros as they were was pointless without
also moving to a new public API. While improving the durability and
readability of the macro code I also rewrote entire sections to accommodate
the new design.
A quick overview of the Matrix protocol for reference: a client sends content
that is interpreted by the server as events. The server distributes those
events out to other clients and other servers (the server case is known as
federation). Ruma groups these events by kind Message, State, Ephemeral,
ToDevice and Basic which are represented as generic structs (StateEvent<C>).
Each event kind needs to be able to hold many different content types, for
state events there is room creation, room name, and membership events to name a
few. Using the macros, enums are generated to represent all state event
possibilities, so a variant for membership, room name, etc. These types exist
to support the core API request and response types for each endpoint that is
defined by the Matrix specification.
GSoC Starts and I start with ruma-events
More work on ruma-events, now a part of the new mono-repo
Work on ruma-client-api
Back to ruma-events
Continuing maintenance
One of my personal goals was to become more familiar with git. With the help
of my mentor I now feel more confident using this tool that is so essential to
developers. I became fairly adept at merging, rebasing, and navigating all the
headaches that come with that. I learned plenty of new commands. A few
highlights: cherry-pick and specific uses of reset to avoid copy-pasting
fixes and adding more commits. I used the reset command to craft good
commits, splitting work into appropriate chunks. I am glad that I had the
opportunity to hone my git skills. I feel like I have accomplished my goal and
then some!
I am proud of the work that I have done: Being part of moving ruma-events much
closer to the 0.22 release and creating macros to generate types specific to
the Matrix specification. Working with the community that has grown around Ruma
has been rewarding and I plan on sticking around.
This week in the ruma/matrix Google Summer of Code project, I worked on refactoring both ruma-api and ruma-events. After moving some of the larger chunks of the ruma_api_macro::api::Api::to_tokens method to helper functions, I spent time removing repetition from the Request/Response code generated by the ruma_api! macro. For ruma-events, the input parsing was changed to only allow valid names for the Any*Event enums. Altering the input parsing had the added benefit of replacing all of the string comparison and manipulation with strongly typed comparison and manipulation.
The final few issues to be resolved before the next crates.io release for ruma-events can happen are related to redacted events. Support for redacted events was added to the Any*Event enums, they now have redacted variants of each event kind. A few follow-up PR's have been merged to fully integrate redacted events into ruma-events, fixing specific event deserialization issues and splitting the UnsignedData struct into Unsigned and RedactedUnsigned.