GNOME

I have started a little experiment in porting bits of the widely-used bzip2/bzlib to Rust. I hope this can serve to refresh bzip2, which had its last release in 2010 and has been nominally unmaintained for years. I hope to make several posts ... [More] detailing how this port is done. In this post, I'll talk about setting up a Rust infrastructure for bzip2 and my experiments in replacing the C code that does CRC32 computations. Super-quick summary of how librsvg was ported to Rust Add the necessary autotools infrastructure to build a Rust sub-library that gets linked into the main public library. Port bit by bit to Rust. Add unit tests as appropriate. Refactor endlessly. MAINTAIN THE PUBLIC API/ABI AT ALL COSTS so callers don't notice that the library is being rewritten under their feet. I have no idea of how bzip2 works internally, but I do know how to maintain ABIs, so let's get started. Bzip2's source tree As a very small project that just builds a library and couple of executables, bzip2 was structured with all the source files directly under a toplevel directory. The only tests in there are three reference files that get compressed, then uncompressed, and then compared to the original ones. As the rustification proceeds, I'll move the files around to better places. The scheme from librsvg worked well in this respect, so I'll probably be copying many of the techniques and organization from there. Deciding what to port first I looked a bit at the bzip2 sources, and the code to do CRC32 computations seemed isolated enough from the rest of the code to port easily. The CRC32 code was arranged like this. First, a lookup table in crc32table.c: UInt32 BZ2_crc32Table[256] = { 0x00000000L, 0x04c11db7L, 0x09823b6eL, 0x0d4326d9L, 0x130476dcL, 0x17c56b6bL, 0x1a864db2L, 0x1e475005L, ... } And then, three macros in bzlib_private.h which make up all the CRC32 code in the library: extern UInt32 BZ2_crc32Table[256]; #define BZ_INITIALISE_CRC(crcVar) \ { \ crcVar = 0xffffffffL; \ } #define BZ_FINALISE_CRC(crcVar) \ { \ crcVar = ~(crcVar); \ } #define BZ_UPDATE_CRC(crcVar,cha) \ { \ crcVar = (crcVar << 8) ^ \ BZ2_crc32Table[(crcVar >> 24) ^ \ ((UChar)cha)]; \ } Initially I wanted to just remove this code and replace it with one of the existing Rust crates to do CRC32 computations, but first I needed to know which variant of CRC32 this is. Preparing the CRC32 port so it will not break I needed to set up tests for the CRC32 code so the replacement code would compute exactly the same values as the original: Rename crc32table.c to crc32.c - that file is going to hold all the CRC32 code, not only the lookup table. Turn the CRC32 macros into functions - so I can move them to Rust and have the C code call them. Then I needed a test that computed the CRC32 values of several strings, so I could capture the results and make them part of the test. static const UChar buf1[] = ""; static const UChar buf2[] = " "; static const UChar buf3[] = "hello world"; static const UChar buf4[] = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, "; int main (void) { printf ("buf1: %x\n", crc32_buffer(buf1, strlen(buf1))); printf ("buf2: %x\n", crc32_buffer(buf2, strlen(buf2))); printf ("buf3: %x\n", crc32_buffer(buf3, strlen(buf3))); printf ("buf4: %x\n", crc32_buffer(buf4, strlen(buf4))); // ... } This computes the CRC32 values of some strings using the original algorithm, and prints their results. Then I could cut&paste those results, and turn the printf into assert — and that gives me a test. int main (void) { assert (crc32_buffer (buf1, strlen (buf1)) == 0x00000000); assert (crc32_buffer (buf2, strlen (buf2)) == 0x29d4f6ab); assert (crc32_buffer (buf3, strlen (buf3)) == 0x44f71378); assert (crc32_buffer (buf4, strlen (buf4)) == 0xd31de6c9); // ... } Setting up a Rust infrastructure for bzip2 Two things made this reasonably easy: A patch from Stanislav Brabec, from Suse, to add an autotools framework to bzip2 The existing Autotools + Rust machinery in librsvg. I.e. "copy and paste from somewhere that I know works well". Wonderful! This is the commit that adds a Rust infrastructure for bzip2. It does the following: Create a Cargo workspace (a Cargo.toml in the toplevel) with a single member, a bzlib_rust directory where the Rustified parts of the code will live. Create bzlib_rust/Cargo.toml and bzlib_rust/src for the Rust sources. This will generate a staticlib for libbzlib_rust.a, that can be linked into the main libbz2.la. Puts in automake hooks so that make clean, make check, etc. all do what you expect for the Rust part. As a side benefit, librsvg's Autotools+Rust infrastructure already handled things like cross-compilation correctly, so I have high hopes that this will be good enough for bzip2. Can I use a Rust crate for CRC32? There are many Rust crates to do CRC computations. I was hoping especially to be able to use crc32fast, which is SIMD-accelerated. I wrote a Rust version of the "CRC me a buffer" test from above to see if crc32fast produced the same values as the C code, and of course it didn't. Eventually, after asking on Mastodon, Kepstin figured out what variant of CRC32 is being used in the original code. It turns out that this is directly doable in Rust with the git version of the crc crate. This crate lets one configure the CRC32 polynomial and the mode of computation; there are many variants of CRC32 and I wasn't fully aware of them. The magic incantation is this: let mut digest = crc32::Digest::new_custom(crc32::IEEE, !0u32, !0u32, crc::CalcType::Normal); With that, the Rust test produces the same values as the C code. Yay! But it can't be that easy Bzlib stores its internal state in the EState struct, defined in bzlib_private.h. That struct stores several running CRC32 computations, and the state for each one of those is a single UInt32 value. However, I cannot just replace those struct fields with something that comes from Rust, since the C code does not know the size of a crc32::Digest from Rust. The normal way to do this (say, like in librsvg) would be to turn UInt32 some_crc into void *some_crc and heap-allocate that on the Rust side, with whatever size it needs. However! It turns out that bzlib lets the caller define a custom allocator so that bzlib doesn't use plain malloc() by default. Rust lets one define a global, custom allocator. However, bzlib's concept of a custom allocator includes a bit of context: typedef struct { // ... void *(*bzalloc)(void *opaque, int n, int m); void (*bzfree)(void *opaque, void *ptr); void *opaque; } bz_stream; The caller sets up bzalloc/bzfree callbacks and an optional opaque context for the allocator. However, Rust's GlobalAlloc is set up at compilation time, and we can't pass that context in a good, thread-safe fashion to it. Who uses the bzlib custom allocator, anyway? If one sets bzalloc/bzfree to NULL, bzlib will use the system's plain malloc()/free() by default. Most software does this. I am looking in Debian's codesearch for where bzalloc gets set, hoping that I can figure out if that software really needs a custom allocator, or if they are just dressing up malloc() with logging code or similar (ImageMagick seems to do this; Python seems to have a genuine concern about the Global Interpreter Lock). Debian's codesearch is a fantastic tool! The first rustified code I cut&pasted the CRC32 lookup table and fixed it up for Rust's syntax, and also ported the CRC32 computation functions. I gave them the same names as the original C ones, and exported them, e.g. const TABLE: [u32; 256] = [ 0x00000000, 0x04c11db7, 0x09823b6e, 0x0d4326d9, ... }; #[no_mangle] pub unsafe extern "C" fn BZ2_update_crc(crc_var: &mut u32, cha: u8) { *crc_var = (*crc_var << 8) ^ TABLE[((*crc_var >> 24) ^ u32::from(cha)) as usize]; } This is a straight port of the C code. Rust is very strict about integer sizes, and arrays can only be indexed with a usize, not any random integer — hence the explicit conversions. And with this, and after fixing the linkage, the tests pass! First pass at rustifying CRC32: done. But that does one byte at a time Indeed; the original C code to do CRC32 only handled one byte at a time. If I replace this with a SIMD-enabled Rust crate, it will want to process whole buffers at once. I hope the code in bzlib can be refactored to do that. We'll see! How to use an existing Rust crate for this I just found out that one does not in fact need to use a complete crc32::Digest to do equivalent computations; one can call crc32::update() by hand and maintain a single u32 state, just like the original UInt32 from the C code. So, I may not need to mess around with a custom allocator just yet. Stay tuned. In the meantime, I've filed a bug against crc32fast to make it possible to use a custom polynomial and order and still get the benefits of SIMD. [Less]

Between my last blog post and now I have been accepted into the Google Summer of Code programme (or GSoC for short), which is basically an opportunity for students to work on open-sorce software projects throughout the summer and also receive a ... [More] stipend (money) for it. The coding period officially began on the 27th of May, but I actually started working on my project before that in order to get a head start. Throughout the summer I will be implementing a feature for GNOME Games. To be more specific, I will be implementing a “Savestates Manager”. The feature itself has already been designed and the details about how it should work are explained very well in this wiki page: https://wiki.gnome.org/Design/Playground/Games/Snapshots UI Design for the Savestates Manager Currently I’m working on automatically migrating files from the old directory layout to the new one in order to support multiple savestates. This is how the new directory layout is going to look like: savestates/ [platform]-[uid]-[core]/ [date]/ snapshot screenshot save save dir/ media The snapshot file is probably the most important one because it’s used to set the state of the emulator core. The creation date is used to uniquely identify savestates. I will keep adding blog posts as I continue to make progress and hopefully there will also be some UI to screenshot before the next post. [Less]

Being an #old newcomer myself , I am still lacking here and there about Gnome , So just few weeks ago I was trying to learn more about our Gitlab instance, thanks to csoriano for not just letting me know about my query but also for letting me help ... [More] him in this newcomer initiative. The main motto is:- “Remove every possible obstacle in the way of a newcomer’s first contribution” So basically Carlos encouraged me that he also wants someone who has experienced this newcomer journey recently which can help him know what can be improved. I really like the fact that even though being just 8 months in Gnome I can work with the President of the Gnome Project itself , this is what I love about open source. So basically when Carlos got time out of his busy schedule , I was able to discuss few points with him which we can improve:- 1. Remove Bugzilla links and mentions from Gnome wiki 2. It will be cool to have a project recommendation system:- whatcanidoforfedora.org 3. Focus on IRC channels where people can get help etc. I also proposed the idea of creation of sudo pet projects for example :- https://github.com/ibqn/libgd-vala-samples or mention the existing pet projects which developers use to experiment with:- https://gitlab.gnome.org/exalm/sidebarexample https://gitlab.gnome.org/albfan/textviewhyperlinks [This one was my starting point] The point of sudo projects is that , for newcomers it’s really hard to straightaway jump into huge code of projects where they might not even know things like vala , gtk , code-style , … So these pet projects I believe that can help newcomers play around with the technology and make them learn how workflow goes in Gnome about gitlab, coding style etc. Carlos liked the idea but it’s complex , hence for now main focus is to improve the guides and make sure they are up to date and ensure they work. If any newcomer wants to give more feedback or maybe any existing developer wants to give feedback from his past journey kindly please do in #newcomers For example exalm gave:-  “A common obstacle is that people clone projects before forking them on gitlabmake changes and then are told that actually they needed to fork to do a MR” Here we need to identify things which we can improve and innovate so that our newcomers can easily get on board in the community for contributions. At last I will say, although I am in GSOC 2019, but I really love the fact that there are so many ways in which I can contribute to this lovely community and one of them is this, finally thanks to Carlos who get me involved in this to help him . Work Done Till Now:- 1.For starting I took it upon myself and tried to replace Bugzilla from the newcomers guide with Issue tracker at Gitlab. Although I removed it as much as I can , but still if anyone finds mention of Bugzilla where it’s highly focused upon in the newcomers guide please help in the improvement. [Less]

Being an #old newcomer myself , I am still lacking here and there about Gnome , So just few weeks ago I was trying to learn more about our Gitlab instance, thanks to csoriano for not just letting me know about my query but also for letting me help ... [More] him in this newcomer initiative. The main motto is:- “Remove every possible obstacle in the way of a newcomer’s first contribution” So basically Carlos encouraged me that he also wants someone who has experienced this newcomer journey recently which can help him know what can be improved. I really like the fact that even though being just 8 months in Gnome I can work with the President of the Gnome Project itself , this is what I love about open source. So basically when Carlos got time out of his busy schedule , I was able to discuss few points with him which we can improve:- 1. Remove Bugzilla links and mentions from Gnome wiki 2. It will be cool to have a project recommendation system:- whatcanidoforfedora.org 3. Focus on IRC channels where people can get help etc. I also proposed the idea of creation of sudo pet projects for example :- https://github.com/ibqn/libgd-vala-samples or mention the existing pet projects which developers use to experiment with:- https://gitlab.gnome.org/exalm/sidebarexample https://gitlab.gnome.org/albfan/textviewhyperlinks [This one was my starting point] The point of sudo projects is that , for newcomers it’s really hard to straightaway jump into huge code of projects where they might not even know things like vala , gtk , code-style , … So these pet projects I believe that can help newcomers play around with the technology and make them learn how workflow goes in Gnome about gitlab, coding style etc. Carlos liked the idea but it’s complex , hence for now main focus is to improve the guides and make sure they are up to date and ensure they work. If any newcomer wants to give more feedback or maybe any existing developer wants to give feedback from his past journey kindly please do in #newcomers For example exalm gave:-  “A common obstacle is that people clone projects before forking them on gitlabmake changes and then are told that actually they needed to fork to do a MR” Here we need to identify things which we can improve and innovate so that our newcomers can easily get on board in the community for contributions. At last I will say, although I am in GSOC 2019, but I really love the fact that there are so many ways in which I can contribute to this lovely community and one of them is this, finally thanks to Carlos who get me involved in this to help him . Work Done Till Now:- 1.For starting I took it upon myself and tried to replace Bugzilla from the newcomers guide with Issue tracker at Gitlab. Although I removed it as much as I can , but still if anyone finds mention of Bugzilla where it’s highly focused upon in the newcomers guide please help in the improvement. [Less]

Contributing to GNOME Boxes GNOME Boxes  is used to view, access, and manage remote and virtual machines. Currently it is able to do either express-installations on a downloaded ISO or to download an ISO and offer the option to express-install ... [More] it. I stumbled upon the project ‘Improve GNOME Boxes express-installations by adding support to tree-based installations’, it required the knowledge of Object Oriented programming and I had done this course last semester only and had a project for the course. Even though I had used Java primarily for the project, it didn’t take me long to understand Vala. I quickly jumped into the irc channel of GNOME Boxes, introduced myself to the mentors(one of them being Felipe Borges), got the source code and built it. Initially it took me some time to understand how things were working but the mentors were patient enough to even answer my dumbest queries. I then started working on a few bugs, they helped me understand the front-end part of the project. I’ve fixed around 6 issues till now and am now working on understanding how the back-end works, mainly how we use libosinfo to get certain information about operating systems, hypervisors and the (virtual) hardware devices they can support.Happy Coding :) [Less]

Contributing to GNOME Boxes GNOME Boxes  is used to view, access, and manage remote and virtual machines. Currently it is able to do either express-installations on a downloaded ISO or to download an ISO and offer the option to express-install ... [More] it. While going through the projects listed for Outreachy May to August 2019 Internships, I stumbled upon this project ‘Improve GNOME Boxes express-installations by adding support to tree-based installations’, it required the knowledge of Object Oriented programming and I had done this course last semester only and had a project for the course. Even though I had used Java primarily for the project, it didn’t take me long to understand Vala. I quickly jumped into the irc channel of GNOME Boxes, introduced myself to the mentors(one of them being Felipe Borges), got the source code and built it. Initially it took me some time to understand how things were working but the mentors were patient enough to even answer my dumbest queries. I then started working on a few bugs, they helped me understand the front-end part of the project. I’ve fixed around 6 issues till now and am now working on understanding how the back-end works, mainly how we use libosinfo to get certain information about operating systems, hypervisors and the (virtual) hardware devices they can support.Happy Coding :) [Less]

Hello Everyone, I am really excited to share with you all that this summer I will be working full-time with Gnome on the project Implement side-by side diff view on the Gitg application. I am really grateful to the community who considered me ... [More] the right person for the job and gave me this wonderful opportunity. I believe in learning and believe me I have never learned anywhere better than being here in GNOME from my last 8 months of journey here. I have learnt things like:- Understanding Code base Basic git skills Communication skills Managing time Reading Docs etc. These are just a few to name and also few in quantity, a whole mountain of things still remain which I believe I will be getting to learn this summer with Gnome. I can’t thank enough to Google as well who introduced and sponsored this program. The most important part I like here was that Google gave organisations freedom to choose the project and the students which to me seems like that Google really respects the freedom of Organisations and their vision. So kudos to Google I know maybe I am a little low on skills for the project but my motive always comes up if my mentor trust me and I believe he knows my capabilities more than I know myself. I really look forward to work with Albfan and I believe that out of any other benefit which GSOC gives to their students this is the most precious one. I have so many people to thank for this : Csoriano, Exalm, Andre, Chergert, Bil, Albfan, ….. If I missed any one’s name here’s the final:- Developers of GNOME It’s my great fortune to be part of this family. Thanks [Less]

Hello Everyone, I am really excited to share with you all that this summer I will be working full-time with Gnome on the project Implement side-by side diff view on the Gitg application. I am really grateful to the community who considered me ... [More] the right person for the job and gave me this wonderful opportunity. I believe in learning and believe me I have never learned anywhere better than being here in GNOME from my last 8 months of journey here. I have learnt things like:- Understanding Code base Basic git skills Communication skills Managing time Reading Docs etc. These are just a few to name and also few in quantity, a whole mountain of things still remain which I believe I will be getting to learn this summer with Gnome. I can’t thank enough to Google as well who introduced and sponsored this program. The most important part I like here was that Google gave organisations freedom to choose the project and the students which to me seems like that Google really respects the freedom of Organisations and their vision. So kudos to Google I know maybe I am a little low on skills for the project but my motive always comes up if my mentor trust me and I believe he knows my capabilities more than I know myself. I really look forward to work with Albfan and I believe that out of any other benefit which GSOC gives to their students this is the most precious one. I have so many people to thank for this : Csoriano, Exalm, Andre, Chergert, Bil, Albfan, ….. If I missed any one’s name here’s the final:- Developers of GNOME It’s my great fortune to be part of this family. Thanks [Less]

Hi ! Yeti here, I’m currently an engineering student and during my free time I contribute to this cool open-source app called GNOME Games (or just Games for short). One would expect an app with that name to be some kind of collection of mini ... [More] games for GNOME but you would be wrong. Games markets itself as a “games manager”, which means it automatically detects all of the games installed on your machine and lists them, such that you have all of your games nicely gathered together in one window. Games Collection view (image taken from the Big Net) But IMO (“in my opinion” in IRC jargon) the juiciest feature of this app is that it comes packed with a bunch of emulator cores which means that you can run retro games like Game Boy games directly in the Games app window. My favorite retro game, ‘Asteroids’ In this blog I will post about my various endeavors in contributing to this app. [Less]

Thanks for joining me! Good company in a journey makes the way seem shorter. — Izaak Walton                   This blog is gonna be about my journey of Google Summer of Code.  It was just few months ago, when I had no idea what GSoC is or how open ... [More] source development works. Randomly, I stumbled upon youtube talks about open source development. The scale and boldness of OSDG really intrigued me. My whole life I have always been fascinated by new ideas or technologies, and it is my biggest dream to be part of such flow. I always like to challenge myself with new problems and tasks. So I dived into world of open source.  When I decided I wanted to contribute to some org, I started looking for different types of orgs, I came across many orgs which sounded interesting. One of them was GNOME. I have been GNOME user since I started using laptops. I have been fascinated by the vast scale of GNOME applications. I definitely wanted to contribute something to this org. I started exploring different projects within GNOME, since all of them were interesting, I filtered projects by the technology I am most comfortable with.                          I have great experience working with Javascript so I went ahead with GNOME Maps, I started looking into newcomer issues. It was around this time, I realised, there’s something called Google Summer of Code, a great opportunity for student developers to explore the realm of open source development.  I was very excited to hear about this opportunity.   [Less]