I Use This!
Very High Activity

News

Analyzed 2 days ago. based on code collected about 1 month ago.
Posted 2 days ago
TLDR: in Endless OS, we switched the IO scheduler from CFQ to BFQ, and set the IO priority of the threads doing Flatpak downloads, installs and upgrades to “idle”; this makes the interactive performance of the system while doing Flatpak operations ... [More] indistinguishable from when the system is idle. At Endless, we’ve been vaguely aware for a while that trying to use your computer while installing or updating apps is a bit painful, particularly on spinning-disk systems, because of the sheer volume of IO performed by the installation/update process. This was never particularly high priority, since app installations are user-initiated, and until recently, so were app updates. But, we found that users often never updated their installed apps, so earlier this year, in Endless OS 3.3.10, we introduced automatic background app updates to help users take advantage of “new features and bug fixes” (in the generic sense you so often see in iOS/Android app release notes). This fixed the problem of users getting “stuck” on old app versions, but made the previous problem worse: now, your computer becomes essentially unusable at arbitrary times when app updates happen. It was particularly bad when users unboxed a system with an older version of Endless OS (and hence a hundred or so older apps) pre-installed, received an automatic OS update, then rebooted into a system that’s unusable until all those apps have been updated. At first, I looked for logic errors in (our versions of) GNOME Software and Flatpak that might cause unneccessary IO during app updates, without success. We concluded that heavy IO load when updating a large app or runtime is largely unavoidable,1 so I switched to looking at whether we could mitigate this by tweaking the IO scheduler. The BFQ IO scheduler is supposed to automatically prioritize interactive workloads over bulk workload, which is pretty much exactly what we’re trying to do. The specific example its developers give is watching a video, without hiccups, while copying a huge file in the background. I spent some time with the BFQ developers’ own suite of benchmarks on two test systems: a Lenovo Yoga 900 (with an Intel i5-6200U @ 2.30GHz and a consumer-grade M.2 SSD) and an Endless Mission One (an older system with a Celeron CPU and a laptop-class spinning disk). Neither JP nor I were able to reproduce any interesting results for the dropped-frames benchmark: with either BFQ or CFQ (the previous default IO scheduler), the Yoga essentially never dropped frames, whereas the IO workloads immediately rendered the Mission totally unusable. I had rather more success with a benchmark which measures the time to launch LibreOffice: On the Yoga, when the system was idle, the mean launch time went from 2.838s under CFQ to 2.98s under BFQ (a slight regression), but with heavy background IO, the mean launch time went from 16s with CFQ (standard deviation 0.11) to 3s with BFQ (standard deviation 0.51). On the Mission, with modest background IO, the mean launch time was 108 seconds under BFQ, which sounds awful; but under CFQ, I gave up waiting for LibreOffice to start after 8 minutes! Emboldened by these results, I went on to look at how the same “time to launch LibreOffice” benchmark fared when the background IO load is “installing and uninstalling a Lollipop Flatpak bundle in a loop”. I also looked at using ionice -c3 to set the IO priority of the install/uninstall loop to idle, which does what its name suggests: BFQ essentially will never serve IO at the idle priority if there is IO pending at any higher priority. You can see some raw data or look at some extended discussion copied from our internal issue tracker to a Flatpak pull request, but I suggest just looking at this chart: What does it all mean? The coloured bars represent median launch time in seconds for LibreOffice, across 15/30 trials for Yoga/Mission respectively. The black whiskers show the minimum and maximum launch times observed. I know this should have been a box-and-whiskers or violin plot, but I realised too late that multitime does not give enough information to draw those. “unloaded” refers to the performance when the system is otherwise idle. “shell-loop” refers to running while true; do flatpak install -y /home/wjt/Downloads/org.gnome.Lollypop.flatpak; flatpak uninstall -y org.gnome.Lollypop/x86_64/stable; done; “long-lived” refers to performing the same operations with the Flatpak API in a long-lived process. I tried this because I understood that BFQ gives new processes a slight performance boost, but on a real system the GNOME Software and Flatpak system helper processes are long-lived. As you can see, the behaviour under BFQ is actually the other way around in the worst case, and identical for CFQ and in the median case. The “ionice-” prefix means the Flatpak operation was run under ionice -c3. Switching from CFQ to BFQ makes the worst case a little worse at the default IO priority, but the median case much better. Setting the IO priority of the Flatpak process(es) to idle erases that worst-case regression under BFQ, and dramatically improves the median case under CFQ. In combination, the time to launch LibreOffice while performing Flatpak operations in the background on the Mission went from 24 seconds to 12 seconds by switching to BFQ & setting the IO priority to idle. So, by switching to BFQ and setting IO priorities appropriately, the system’s interactive performance while performing background updates is now essentially indistinguishable from when the system is idle. To implement this in practice, Rob McQueen wrote some patches to set the IO priority of the Flatpak system helper and GNOME Software’s worker threads to idle (both changes are upstream) and changed Endless OS’s default IO scheduler to BFQ where available. As Matthias put it on #flatpak when shown this chart and that first link: “not bad for a 1-line change”. Of course, this means apps take a bit longer to install, even on a mostly-idle system. No, I don’t have numbers on how big the impact is: this work happened months ago and it’s taken me this long to write it up because I couldn’t find the time to collect more data. But my colleague Umang is working on eliminating up to half of the disk IO performed during Flatpak installations so that should more than make up for it! modulo Umang’s work mentioned in the coda [Less]
Posted 2 days ago
It happens sneakily, but there’s more things going on in the Tracker front than the occasional fallout. Yesterday 2.2.0-alpha1 was released, containing some notable changes. On and off during the last year, I’ve been working on a massive rework of ... [More] the SPARQL parser. The current parser was fairly solid, but hard to extend for some of the syntax in the SPARQL 1.1 spec. After multiple attempts and failures at implementing property paths, I convinced myself this was the way forward. The main difference is that the previous parser was more of a serializer to SQL, just minimal state was preserved across the operation. The new parser does construct an expression tree so that nodes may be shuffled/reevaluated. This allows some sweet things: Property paths are a nice resource to write more idiomatic SPARQL, most property path operators are within reach now. There’s currently support for sequence paths: # Get all files in my homedir SELECT ?elem { ?elem nfo:belongsToContainer/nie:url 'file:///home/carlos' } And inverse paths: # Get all files in my homedir by inverting # the child to container relation SELECT ?elem { ?homedir nie:url 'file:///home/carlos' ; ^nfo:belongsToContainer ?elem } There’s harder ones like + and * that will require recursive selects, and there’s the negation (!) operator which is not possible to implement yet. We now have prepared statements! A TrackerSparqlStatement object was introduced, capable of holding a query with parameters which can be set/replaced prior to execution. conn = tracker_sparql_connection_get (NULL, NULL); stmt = tracker_sparql_connection_query_statement (conn, "SELECT ?u { ?u fts:match ~term }", NULL, NULL); tracker_sparql_statement_bind_string (stmt, "term", search_term); cursor = tracker_sparql_statement_execute (stmt, NULL, NULL); This is a long sought protection for injections. The object is cacheable and can service multiple cursors asynchronously, so it will also be an improvement for frequent queries. More concise SQL is generated at places, which brings slight improvements on SQLite query planning. This also got the ideas churning towards future plans, the trend being a generic triple store as much sparql1.1 capable as possible. There’s also some ideas about better data isolation for Flatpak and sandboxes in general (seeing the currently supported approach didn’t catch on). Those will eventually happen in this or following cycles, but I’ll reserve that for other blog post. An eye was kept on memory usage too (mostly unrealized ideas from the performance hackfest earlier this year), tracker-store has been made to automatically shutdown when unneeded (ideally most of the time, since it just takes care of updates and the unruly apps that use the bus connection), and tracker-miner-fs took over the functionality of tracker-miner-apps. That’s 2 processes less in your default session. In general, we’re on the way to an exciting release, and there’s more to come! [Less]
Posted 3 days ago by nore...@blogger.com (Robert Ancell)
I've been spending a bit of time recently working on GNOME Settings. One part of this has been bringing some of the older panel code up to modern standards, one of which is making use of GtkBuilder templates. I wondered if any of these changes ... [More] would show in the stats, so I wrote a program to analyse each branch in the git repository and break down the code between C and GtkBuilder. The results were graphed in Google Sheets: This is just the user accounts panel, which shows some of the reduction in C code and increase in GtkBuilder data: Here's the breakdown of which panels make up the codebase: I don't think this draws any major conclusions, but is still interesting to see. Of note: Some of the changes make in 3.28 did reduce the total amount of code! But it was quickly gobbled up by the new Thunderbolt panel. Network and Printers are the dominant panels - look at all that code! I ignored empty lines in the files in case differing coding styles would make some panels look bigger or smaller. It didn't seem to make a significant difference. You can see a reduction in C code looking at individual panels that have been updated, but overall it gets lost in the total amount of code. I'll have another look in a few cycles when more changes have landed (I'm working on a new sound panel at the moment). [Less]
Posted 3 days ago
As discussed in my previous blog post one of my TODO list items for plymouth is creating a new plymouth theme.Since the transition to plymouth is not entirely smooth plymouth by default will wait 5 seconds (counted from starting the kernel) before ... [More] showing itself so that on systems which boot under 5 seconds it never shows. As can be seen in this video, this leads to a very non-smooth experience when the boot takes say 7 seconds as plymouth then only shows briefly, leading to a kinda "flash" effect while it briefly shows.Another problem with the 5 second wait, is now that we do not show GRUB the user is looking at the firmware's bootsplash for not only the often long firmware initialization time, but also for the 5 seconds plymouth waits on top, making it look as if nothing is happening.To fix this I've been working on a new plymouth theme which draws a spinner over the firmware boot splash, eliminating the ugly transition from the firmware boot splash to plymouth. This also allows removing the show-delay, so that we provide feedback that something is happening as soon as plymouth starts.Firmware being firmware getting this done right was somewhat harder then I expected, but I've a first "draft" of a new theme doing this now. I've created some videos showing 2 different systems booting the new theme: My workstation booting, plain version of the theme My workstation booting, Fedora branded version of the theme My laptop with diskcrypt booting, plain version of the theme My laptop with diskcrypt booting, Fedora branded version of the theme Note the videos with diskcrypt where paused when I entered my passphrase. So there is a bit of a jump in them because of this.I've built a test version of plymouth for Fedora 29, to give this a try download all rpm files from here except the .src.rpm and -devel files and then from a directory with all those files in it, run:sudo rpm -Uvh plymouth*.rpmSince plymouth is part of your initrd, you also need to regenerate your initrd:sudo dracut -f /boot/initramfs-$(uname -r).img $(uname -r)This regenerates the initrd for the kernel you are currently running, so if you've installed a kernel update and have not rebooted since then you may not get the new theme when rebooting. In this case rerun the dracut command after rebooting.Now reboot and you should get the new spinner on firmware-boot-splash theme, with Fedora branding.If you give this a try and the new theme somehow does not look correct, please mail at hdegoede@redhat.com. If you mail me about the theme not displaying correctly please attach the /run/plymouth.log file which this test-build generates to the email and a video of how the theme misbehaves would be great too.I still need to discuss the idea of using a new theme incorporating the firmware boot splash with the GNOME design team so this is all subject to change. [Less]
Posted 3 days ago
As discussed in my previous blog post one of my TODO list items for plymouth is creating a new plymouth theme.Since the transition to plymouth is not entirely smooth plymouth by default will wait 5 seconds (counted from starting the kernel) before ... [More] showing itself so that on systems which boot under 5 seconds it never shows. As can be seen in this video, this leads to a very non-smooth experience when the boot takes say 7 seconds as plymouth then only shows briefly, leading to a kinda "flash" effect while it briefly shows.Another problem with the 5 second wait, is now that we do not show GRUB the user is looking at the firmware's bootsplash for not only the often long firmware initialization time, but also for the 5 seconds plymouth waits on top, making it look as if nothing is happening.To fix this I've been working on a new plymouth theme which draws a spinner over the firmware boot splash, eliminating the ugly transition from the firmware boot splash to plymouth. This also allows removing the show-delay, so that we provide feedback that something is happening as soon as plymouth starts.Firmware being firmware getting this done right was somewhat harder then I expected, but I've a first "draft" of a new theme doing this now. I've created some videos showing 2 different systems booting the new theme: My workstation booting, plain version of the theme My workstation booting, Fedora branded version of the theme My laptop with diskcrypt booting, plain version of the theme My laptop with diskcrypt booting, Fedora branded version of the theme Note the videos with diskcrypt where paused when I entered my passphrase. So there is a bit of a jump in them because of this.I've built a test version of plymouth for Fedora 29, to give this a try download all rpm files from here except the .src.rpm and -devel files and then from a directory with all those files in it, run:sudo rpm -Uvh plymouth*.rpmSince plymouth is part of your initrd, you also need to regenerate your initrd:sudo dracut -f /boot/initramfs-$(uname -r).img $(uname -r)This regenerates the initrd for the kernel you are currently running, so if you've installed a kernel update and have not rebooted since then you may not get the new theme when rebooting. In this case rerun the dracut command after rebooting.Note if you've previously followed my instructions to test flickerfree boot, then you need to remove "plymouth.splash_delay=20" from your kernel commandline, since we now no longer want to have a splash-delay.Now reboot and you should get the new spinner on firmware-boot-splash theme, with Fedora branding.If you give this a try and the new theme somehow does not look correct, please mail at hdegoede@redhat.com. If you mail me about the theme not displaying correctly please attach the /run/plymouth.log file which this test-build generates to the email and a video of how the theme misbehaves would be great too.I still need to discuss the idea of using a new theme incorporating the firmware boot splash with the GNOME design team so this is all subject to change. [Less]
Posted 3 days ago
I want a lot to write more in this blog. There are matters I didn’t talk enough about SuperSEC or GUADEC conferences, some announce for 2019 and some some activities in Wikipedia (specially in the Wikiproyecto-Almería and my firsts step in the ... [More] amazing world of SPARQL), less important but I really enjoy. But now I want to keep record of significant advances in the university degree I’m finishing these months. I decided to finish a pending course with special interest in the required degree final work, to work in things I’ve been interested since 2003 but never had the oportunity to focus in deep enough to study, learn and write some useful, I hope, tools. And it’s being fun :-) So now I can say the project blueprint has been approved by the university. It’s named «Creation of artifacts for adoption of ISO/IEC 29110 standards» (document in Spanish, sorry) and the goals are to produce a set of opensource artifacts for the adoption of the 29110 family of standards focused on a light software engineering methodology suitable to be adopted by very small entities (VSEs). At the moment my main target is to work in the «Part 5-4: Agile software development guidelines», currently on development by WG24, using the EPF Composer tool. As a working tool I’m making a (half backed and maybe temporal) website to keep record of related materials at http://29110.olea.org. Hope to announce related news in the next weeks. [Less]
Posted 3 days ago by theaw...@gmail.com (Abhinav Singh)
Sorry for the long wait! In this post I’ll be talking about my visit to the GSoC Mentors Summit 2018. I represented GNOME, sadly alone because the other selected mentor didn’t get the US visa in time. This was my first trip out of India and I ... [More] couldn’t plan it properly1, so I went there for just the two conference days. Meets The [un]conference was a great place to learn about other open source organizations. I was able to meet a lot of people, and I remember talking to folks from HPX, scilab, WorldBrain2, Xwiki, coala, OSGeo3, Java PathFinder, PollyLabs4, KDE, fossasia, Oppia, MuseScore, CiviCRM, and meeting at least two other Abhinavs at the summit. GSoC mentors from over 40 countries! Sessions A wide range of topics were covered in the sessions - funding, licensing, documentation, and in general how to improve open source software. Some sessions were more GSoC oriented like how to - retain mentors, select a good project, communicate better, and improve the program. I would like to highlight few sessions that piqued my interest: Google Season of Docs GSoD is a new program (still being brainstormed) in which technical writers would be able to participate and work on documentation. I believe this would be a great initiative and can help newcomers ramp up much faster! Open Source Personal Assistants Mario from fossasia held this session demonstrating the SUSI AI. It was really refreshing to see an open source personal assistant working so well. I wonder why linux distros do not ship with a working personal assistant while Windows and macOS are continuously deepening their AI integration to aid users. I was able to find a SUSI AI client for linux (written in Gtk). We can definitely look into making its integration easier into GNOME - say how about asking the user during the first run if they would like to enable a personal assistant? GSoC Feedback This was my favourite session. The notes for this session can be found here. I could relate the most to the following points given by other mentors: Mentors have difficulties seeing feedback from evaluations Google should consider raising the lowest bar on the stipend Lightning Talks Lightning talks are the quickest way to gain curiosity according to me. I got to know about many projects that I wouldn’t have otherwise. It was amazing to see the influence of open source on health, robotics, humanities, and science! Most presentations talked about their students’ summer work, but I especially loved the one by openSUSE which was about encouraging those students that had their proposals rejected. The slides for all these talks can be found here. Unfortunately, I did not present as I was too scared :( Chocolates The chocolate table was a delight! It was the first and definitely the last time in my life that I had a chocolate made of 100% cocoa. There was one chocolate that smelled like bubblegum, another one that had spices in it, and one that made popping sounds when eaten! Open source doesn’t only bring diversity in people but in chocolates as well :P Thanks GSoC Mentor Summit 2018 was truly an amazing experience! I am grateful to GNOME for selecting me and to Google for running the wonderful GSoC program. I would also like to give special thanks to Alexander Mikhaylenko for helping out in mentoring even though he didn’t sign up for it. read poor work-life balance due to my day job ↩ Oliver - thanks for the seaweed ↩ Jeff - I watched the movie you suggested ↩ these folks do magic with LLVM ↩ [Less]
Posted 4 days ago
We’ve just added an optional feature to fwupd and the LVFS that some people might find useful: The firmware update process can now tell the user how long in seconds the update is going to take. This means that users can know that a dock update might ... [More] take 5 minutes, and so they start the update process before they go to lunch. A UEFI update will require multiple reboots and will take 45 minutes to apply, and so the user will only apply the update at the end of the day rather than losing access to the their computer for nearly an hour. If you want to use this feature there are currently three ways to assign the duration to the update: Changing the value on the LVFS admin console — the component update panel now has an extra input field to enter the duration in seconds Adding a new attribute to the element, for instance: Adding a ‘quirk’ to fwupd, for instance: [DeviceInstanceId=USB\VID_1234&PID_5678] InstallDuration = 40 For updates requiring a reboot the install duration should include the time to POST the system both before and after the update has run, but it can be approximate. Only users running very new versions of fwupd and gnome-software will be shown the install duration, and older versions will be unchanged as the new property will just be ignored. It’s therefore safe to include in all versions of firmware without adding a the dependency on a specific fwupd version. [Less]
Posted 4 days ago
Last weekend I participated in the conference Jesień Linuksowa 2018 in Ustroń which is located in Poland near the Czech and Slovak borders. It was my first time in a country with so much tragic historical experiences. On the other hand, I was ... [More] impressed by the community members and the organization of the event. We celebrated another edition of Linux Autumn in the hotel Gwarek and my post-event wrap up will take into consideration seven basic points: Organizers This time I was accompanied by my friend Ana Garcia, who is a student at the University of Edinburgh and the members of the organization were supportive and kind all the time with us. We felt a warm environment since we arrive at midnight in the middle of the fog. They helped us with our talks and workshops we offer related to parallelization. We meet new Linux friends: Dominik, Rafal, Filip, Mateusz Kita, and Matej from Red Hat.Topics Interesting topics were presented that included Ansible, catching bugs, packaging to Fedora and the innovations of systemd. Here pictured Marcin, Matej, and Zbigniew talks: My participation I did a topic related to introduction to parallel computers and a workshop to use parallel directives in OpenMP such as static, dynamic and guided. Thanks to Ana for being my co-speaker this time. My talk was set to the morning around 10 am, and we did the workshop after lunch and we are glad that all the participants completed successfully the experience. Fedora 29 party We celebrated with a delicious Fedora 29 cake! What are the new features in Fedora 29 were explained, as well as how to create a FAS account and the system badges. I am glad that Fedora Ambassadors from Peru and Poland, as well as Matej from Red Hat Brno met.Food The organization arranged comfort food more than three times a day, we had extra cookies and water between main menus, plus the cake of course! We tasted delicious Poland food! Special Thanks To Filip Kłębczyk not only for being the general organizer but also for giving me one of the best gifts ever! and to Rafał Lużyński for the invitation and the tour around the city. The uniqueness  Retro gaming area was a geeky-genius-fantastic-idea There was a unique collection of games from the 80’s and we were able to have an entertainment with that treasure. [Less]
Posted 5 days ago
Mail; admin; lunch, status report, sync. with Andras.