Tumblelog by Soup.io
Newer posts are loading.
You are at the newest post.
Click here to check if anything new just came in.

May 21 2018

2000 9ba9
Martin Pitt: De-Googling my phone, reloaded

Andy Wingo: correct or inotify: pick one

Let's say you decide that you'd like to see what some other processes on your system are doing to a subtree of the file system. You don't want to have to change how those processes work -- you just want to see what files those processes create and delete.

One approach would be to just scan the file-system tree periodically, enumerating its contents. But when the file system tree is large and the change rate is low, that's not an optimal thing to do.

Fortunately, Linux provides an API to allow a process to receive notifications on file-system change events, called inotify. So you open up the inotify(7) manual page, and are greeted with this:

With careful programming, an application can use inotify to efficiently monitor and cache the state of a set of filesystem objects. However, robust applications should allow for the fact that bugs in the monitoring logic or races of the kind described below may leave the cache inconsistent with the filesystem state. It is probably wise to do some consistency checking, and rebuild the cache when inconsistencies are detected.

It's not exactly reassuring is it? I mean, "you had one job" and all.

Reading down a bit farther, I thought that with some "careful programming", I could get by. After a day of trying, I am now certain that it is impossible to build a correct recursive directory monitor with inotify, and I am not even sure that "good enough" solutions exist.

pitfall the first: buffer overflow

Fundamentally, inotify races the monitoring process with all other processes on the system. Events are delivered to the monitoring process via a fixed-size buffer that can overflow, and the monitoring process provides no back-pressure on the system's rate of filesystem modifications. With inotify, you have to be ready to lose events.

This I think is probably the easiest limitation to work around. The kernel can let you know when the buffer overflows, and you can tweak the buffer size. Still, it's a first indication that perfect is not possible.

pitfall the second: now you see it, now you don't

This one is the real kicker. Say you get an event that says that a file "frenemies.txt" has been created in the directory "/contacts/". You go to open the file -- but is it still there? By the time you get around to looking for it, it could have been deleted, or renamed, or maybe even created again or replaced! This is a TOCTTOU race, built-in to the inotify API. It is literally impossible to use inotify without this class of error.

The canonical solution to this kind of issue in the kernel is to use file descriptors instead. Instead of or possibly in addition to getting a name with the file change event, you get a descriptor to a (possibly-unlinked) open file, which you would then be responsible for closing. But that's not what inotify does. Oh well!

pitfall the third: race conditions between inotify instances

When you inotify a directory, you get change notifications for just that directory. If you want to get change notifications for subdirectories, you need to open more inotify instances and poll on them all. However now you have N2 problems: as poll and the like return an unordered set of readable file descriptors, each with their own ordering, you no longer have access to a linear order in which changes occurred.

It is impossible to build a recursive directory watcher that definitively says "ok, first /contacts/frenemies.txt was created, then /contacts was renamed to /peeps, ..." because you have no ordering between the different watches. You don't know that there was ever even a time that /contacts/frenemies.txt was an accessible file name; it could have been only ever openable as /peeps/frenemies.txt.

Of course, this is the most basic ordering problem. If you are building a monitoring tool that actually wants to open files -- good luck bubster! It literally cannot be correct. (It might work well enough, of course.)

reflections

As far as I am aware, inotify came out to address the needs of desktop search tools like the belated Beagle (11/10 good pupper just trying to get his pup on). Especially in the days of spinning metal, grovelling over the whole hard-drive was a real non-starter, especially if the search database should to be up-to-date.

But after looking into inotify, I start to see why someone at Google said that desktop search was in some ways harder than web search -- I mean we all struggle to find files on our own machines, even now, 15 years after the whole dnotify/inotify thing started. Part of it is that the given the choice between supporting reliable, fool-proof file system indexes on the one hand, and overclocking the IOPS benchmarks on the other, the kernel gave us inotify. I understand it, but inotify still sucks.

I dunno about you all but whenever I've had to document such an egregious uncorrectable failure mode as any of the ones in the inotify manual, I have rewritten the software instead. In that spirit, I hope that some day we shall send inotify to the pet cemetery, to rest in peace beside Beagle.

Lennart Poettering: All Systems Go! 2018 CfP Open

May 19 2018

Eisha Chen-yen-su: Internationalization of Fractal (3rd and last part)

“Tl;dr version”: I’ve finished implementing the i18n of Fractal and I’ve submitted a first French translation of it. With some help from Daniel (my mentor) to complete the integration with the build system, so thank him for that! Here are my merge requests: https://gitlab.gnome.org/World/fractal/merge_requests/105 and https://gitlab.gnome.org/World/fractal/merge_requests/107.

I am going to detail a little bit what was done in order to achieve this.

Integrate gettext-rs to the project

I first added gettext-rs as one of the dependencies of Fractal (in this commit), as I have explained it in the previous articles. Then, I put the initialization of gettext-rs by asking it to look for the locale files in ./fractal-gtk/po (in this commit).

Wrap all the translatable strings in the Rust source files

Then I took quite some time to examine every strings in the source files of the crate fractal-gtk to wrap all messages that would end up in the GUI with gettext (it turned out that I didn’t really need to use ngettext). Wrapping some strings in the format! macro was less obvious to deal with though. For instance:

secondary = format!("You've been invited to join to <b>{}</b> room by <b>{}</b>",
                    room_name, sender_name);

Couldn’t be straightforwardly rewritten like this:

secondary = format!(gettext("You've been invited to join to <b>{}</b> room by <b>{}</b>"),
                    room_name, sender_name);

Because the first argument of format! needs to be a string literal, so I had to use this work around instead:

let sentence_template = gettext("You've been invited to join to <b>{room_name}</b> room by <b>{sender_name}</b>");
secondary = sentence_template.replace("{room_name}", room_name.as_str())
                             .replace("{sender_name}", sender_name.as_str());

I explicitly did what format! would have done here without having the constraint of using a string literal. Because in some languages, the place of {room_name} and {sender_name} can be reversed. See this commit for more details.

Adding the support of gettext within the build system

Next, we needed a way to automatically generate PO and POT files and move MO files to the right place for gettext-rs to read. So I was going to have the meson build system helping me for that. I first added a POTFILES.in file which lists all the files with translatable strings in them and a LINGUAS file which list the languages for which we want to have PO files. And I’ve added a meson.build file and updated to one in the project root to be able to generate the mentioned file: you can run `ninja -C _build fractal-pot` to generate a POT file and `ninja -C _build fractal-update-po` to generate/update PO files. See these commit for more details on: the POTFILE.in, the LINGUAS file and the meson.build files.

Once the ability to generate the locale files implemented, Daniel helped my with this commit that removes the hard-coded path to bind the text domain for gettext. And he added a Spanish translation (see this commit). I also submitted a French translation, see this commit and this one.

After my MR merged, my first task for GSoC is completed!! 🙂

Tobias Mueller: Talking at GPN 2018 in Karlsruhe, Germany
Daniel García Moreno: Stickers in Riot

May 18 2018

Marco Trevisan: Hello Planet GNOME!

Hey guys, although I’ve been around for a while hidden in the patches, some months ago (already!?!) I did my application to join the GNOME Foundation, and few days after – thanks to some anonymous votes – I got approved :), and thus I’m officially part of the family!

So, thanks again, and sorry for my late “hello” 🙂

Zeeshan Ali: Collabora and GStreamer spring in Sweden
7208 97ed
Felipe Borges: Boxes now supports RDP connections
Miguel de Icaza: Startup Improvements in Xamarin.Forms on Android

Alexandre Franke: Fractal hackfest in Strasbourg

As Planet GNOME readers should be aware of by now 😉 we had a Fractal hackfest last week. I organised it with the help of Tobias and I want to thank all the parties that not only made it possible, but also made it an awesome event:

  • Epitech, the software engineering school where we had GUADEC in 2014, provided the venue
  • Purism sponsored the snacks and sent people to attend
  • the GNOME Foundation sponsored travel for some attendees and a dinner
  • AIUS took care of the catering on the first day, and also kindly lent us a kettle and coffee maker.

We were lucky to have Matthew from the core Matrix team with us for half of the event to answer our technical questions and help us steer our ship in the right direction.

I got involved with the discussions on various topics but as these have already been covered at length by the other attendees I will focus on some other stuff I managed to do. The first one is the one I’m the most happy about: I brought Quentin, a local student (sitting next to me on the picture), and he made his first GNOME contribution, which happened to be merge request !100 in Fractal 🎉. I also managed to write some code and make some progress on a few things. I landed the changes on generated avatars for users and rooms that don’t have one set, and started working on a couple of other contributions which should be finished in the near future. Keep an eye on the commit log to see them as they happen! 😊 I’m happy with what we have achieved so far and I’m looking forward 🤩 to more good things coming soon.

Apart from the technical side of things, I also tried to act as a city guide and hope my guests liked the places I took them. I for sure had lots of fun hanging out with all those people!

May 17 2018

Rohit Kaushik: Working on GNOME To Do this Summer
Carlos Garnacho: Performance hackfest

May 16 2018

Tobias Bernard: Banquets and Barbecues

tl;dr: We’re splitting up Fractal into two separate apps: One to replace IRC, the other to replace Telegram.

This is an in-depth post on the thinking behind the split of the Fractal app, which was decided at the hackfest in Strasbourg last week. For more information about the hackfest, have a look at my other blog post.

1-1 woes

One of the biggest problems with Fractal at the moment is that 1-1 messaging is pretty terrible. Since the rooms in the sidebar are sorted by most recent activity, high-traffic public rooms (such as GNOME IRC channels) tend to drown out rooms with less traffic, such as 1-1s and small groups. This is problematic because the signal-to-noise ratio in 1-1 chats and small groups tends to be much higher than in high-traffic public rooms. This leaves the user constantly searching for the rooms they care about, while the rooms they don’t care about are always at the top.

1-1 chats are quickly drowned out by high-traffic public rooms in the sidebar

One way to solve this problem is having a favorites group for “important” rooms. This is a feature Fractal has had for a while, and it does solve some of the problems with a room list sorted purely by recent activity. However, it only works well for rooms that are important over long periods of time, and needs to be managed manually. 1-1 chats are often brief, and there can be many of them in parallel. Putting them in favorites doesn’t make sense in many cases, as it would balloon the size of the favorites group, and require lots of manual work when starting or ending a conversation.

The “obvious” solution would be doing what Riot does: Having a separate group of 1-1 rooms in the sidebar, and thereby keeping the 1-1 conversations in one consistent place. However, this creates more problems than it solves. In practice, it results in multiple groups of arbitrary length competing for real estate in the sidebar. If you have a lot of 1-1s, this means that you’ll be able to see very few rooms (even when most of the 1-1s are old and not relevant at the moment). In Riot, this group is capped at 10 visible rooms by default, but that’s still not great if you only need 2 of them at the moment. The category can be collapsed, but then you can’t see which 1-1s have new messages, and it also means lots of busywork collapsing/expanding the group. Clearly this isn’t an ideal solution, which is why we were very hesitant to go down this path.

Riot’s separate 1-1 category doesn’t really solve the problem, because old 1-1s take up a ton of vertical real estate when it’s expanded

A way out?

As we were discussing this issue over the past few months, I started looking more closely at the way people use different messaging tools. One thing I found puzzling is that despite the fact that Matrix theoretically supports the use cases covered by popular apps like Whatsapp and Telegram, few people are actually using it to replace those apps. Instead, they use it to replace IRC and Slack.

Why? My theory is that most chat rooms fall in one of three categories:
Private Chats, which include 1-1s and small groups; Team Chats, which are larger, but still private and invite-only; and Public Rooms, which are basically like IRC.

Team Chats and Public Rooms share many characteristics: Both have relatively high amounts of traffic, and there’s a lot of noise. The main difference is that Team Chats are private and the members rarely change (e.g. a company’s internal Slack), while Public Rooms can be joined by anyone at any time, and there is no expectation of privacy (e.g. #gnome-hackers on IRC).

However, Private Chats have relatively little in common with the other two categories: They are low-traffic, and have little or no noise. This may sound like a small difference, but I think it’s the reason why 1-1s suck in Fractal/Riot/IRC, and why people aren’t using Matrix to replace Telegram.

The Banquet and the Barbecue

I’ve come to the conclusion that one app can’t cover all the use cases that the Matrix protocol supports, and still provide competitive UX. If you design an app to deal with lots of high-traffic rooms (e.g. Riot as it is today), it will suck for 1-1s, so people will use something else for those. Similarly, Telegram is primarily designed for 1-1s and small groups, which is why it’s a terrible experience if you have many high-traffic groups.

If we want Matrix to succeed as more than an IRC/Slack replacement we need multiple apps, each focusing on a distinct use case. For messaging, I think the most important distinction to make is between what I call the Banquet and the Barbecue.

Slack is one of the most widely used apps covering the Banquet use case

The Banquet is a big, loud place. There are tons of people, and you don’t know many of them. Lots of things are happening all the time, and it’s hard to keep track of everything. This is what Matrix is currently mostly used for. Slack, IRC, and Discord are also all in this category.

iMessage is a good example of an app focused on the Barbecue use case

The Barbecue is at the other end of the spectrum: It’s a calm, private environment where friends, family, co-workers, and other acquaintances hang out. Conversations are mostly between 2 or 3 people, slow, and often very personal. Telegram, Whatsapp, iMessage, Facebook Messenger, and a myriad of other chat apps are optimized for this use case.

Fracturing Fractal

Now, what does this mean for Fractal? After a long discussion on Thursday, we decided to split up Fractal into two separate apps with different interfaces, each containing a subset of the user’s Matrix rooms.

Exactly how rooms will be split between the two apps is not 100% clear yet. 1-1s are clearly Barbecue, public rooms are clearly Banquet, but private groups could go either way. For these cases we may need a way to explicitly move rooms between apps. The distinction should probably be part of the Matrix spec, so the intent for a room to be a Barbecue or Banquet room could be set when creating a room, and persist across devices.

The two apps will share practically all the internals, and even large parts of the interface. However, the split will allow us to do some things differently in each app to optimize the interfaces for the different use cases. Some of the changes we’re considering are a bubble-style message view in the Barbecue app, and more room categories (such as low-priority) in the Banquet app’s sidebar.

For more details on the split have a look at the blog posts by Daniel, Eisha, Julian, and Adrien.

Messages and Discussions

How exactly the apps will be branded (and what will happen to the Fractal name we all love) is still being decided, but there is some consensus to move to GNOME-style generic names. The Barbecue app will almost certainly be called “Messages”. For the Banquet app there’s less agreement, but my current favorite is “Discussions”.

Early-stage mockups showing what the two different apps could look like

The Fractal brand will not go away though: We’re thinking of keeping it around as the name of the community project that develops both GNOME Matrix apps, and/or using it for the backend powering both apps.

There are lots of details to be figured out in this transition, both from a design and an implementation perspective, but I’m very excited about this new direction. If you’d like to join the effort, come talk to us on Matrix.

Note: I have no illusions that this change will magically get everyone to leave Whatsapp/Telegram/iMessage and move to Matrix. In the short term, the goal is simply to make Matrix 1-1s a good experience. That said, if we ever want Matrix to make inroads with the general public, I think a move in this direction is an important precondition.

Andy Wingo: lightweight concurrency in lua

Hello, all! Today I'd like to share some work I have done recently as part of the Snabb user-space networking toolkit. Snabb is mainly about high-performance packet processing, but it also needs to communicate with management-oriented parts of network infrastructure. These communication needs are performed by a dedicated manager process, but that process has many things to do, and can't afford to make blocking operations.

Snabb is written in Lua, which doesn't have built-in facilities for concurrency. What we'd like is to have fibers. Fortunately, Lua's coroutines are powerful enough to implement fibers. Let's do that!

fibers in lua

First we need a scheduling facility. Here's the smallest possible scheduler: simply a queue of tasks and a function to run those tasks.

local task_queue = {}

function schedule_task(thunk)
   table.insert(task_queue, thunk)
end

function run_tasks()
   local queue = task_queue
   task_queue = {}
   for _,thunk in ipairs(queue) do thunk() end
end

For our purposes, a task is just a function that will be called with no arguments.

Now let's build fibers. This is easier than you might think!

local current_fiber = false

function spawn_fiber(fn)
   local fiber = coroutine.create(fn)
   schedule_task(function () resume_fiber(fiber) end)
end

function resume_fiber(fiber, ...)
   current_fiber = fiber
   local ok, err = coroutine.resume(fiber, ...)
   current_fiber = nil
   if not ok then
      print('Error while running fiber: '..tostring(err))
   end
end

function suspend_current_fiber(block, ...)
   -- The block function should arrange to reschedule
   -- the fiber when it becomes runnable.
   block(current_fiber, ...)
   return coroutine.yield()
end

Here, a fiber is simply a coroutine underneath. Suspending a fiber suspends the coroutine. Resuming a fiber runs the coroutine. If you're unfamiliar with coroutines, or coroutines in Lua, maybe have a look at the lua-users wiki page on the topic.

The difference between a fibers facility and just coroutines is that with fibers, you have a scheduler as well. Very much like Scheme's call-with-prompt, coroutines are one of those powerful language building blocks that should rarely be used directly; concurrent programming needs more structure than what Lua offers.

If you're following along, it's probably worth it here to think how you would implement yield based on these functions. A yield implementation should yield control to the scheduler, and resume the fiber on the next scheduler turn. The answer is here.

communication

Once you have fibers and a scheduler, you have concurrency, which means that if you're not careful, you have a mess. Here I think the Go language got the essence of the idea exactly right: Do not communicate by sharing memory; instead, share memory by communicating.

Even though Lua doesn't support multiple machine threads running concurrently, concurrency between fibers can still be fraught with bugs. Tony Hoare's Communicating Sequential Processes showed that we can avoid a class of these bugs by treating communication as a first-class concept.

Happily, the Concurrent ML project showed that it's possible to build these first-class communication facilities as a library, provided the language you are working in has threads of some kind, and fibers are enough. Last year I built a Concurrent ML library for Guile Scheme, and when in Snabb we had a similar need, I ported that code over to Lua. As it's a new take on the problem in a different language, I think I've been able to simplify things even more.

So let's take a crack at implementing Concurrent ML in Lua. In CML, the fundamental primitive for communication is the operation. An operation represents the potential for communication. For example, if you have a channel, it would have methods to return "get operations" and "put operations" on that channel. Actually receiving or sending a message on a channel occurs by performing those operations. One operation can be performed many times, or not at all.

Compared to a system like Go, for example, there are two main advantages of CML. The first is that CML allows non-deterministic choice between a number of potential operations in a generic way. For example, you can construct a operation that, when performed, will either get on one channel or wait for a condition variable to be signalled, whichever comes first. In Go, you can only select between operations on channels.

The other interesting part of CML is that operations are built from a uniform protocol, and so users can implement new kinds of operations. Compare again to Go where all you have are channels, and nothing else.

The CML operation protocol consists three related functions: try which attempts to directly complete an operation in a non-blocking way; block, which is called after a fiber has suspended, and which arranges to resume the fiber when the operation completes; and wrap, which is called on the result of a successfully performed operation.

In Lua, we can call this an implementation of an operation, and create it like this:

function new_op_impl(try, block, wrap)
   return { try=try, block=block, wrap=wrap }
end

Now let's go ahead and write the guts of CML: the operation implementation. We'll represent an operation as a Lua object with two methods. The perform method will attempt to perform the operation, and return the resulting value. If the operation can complete immediately, the call to perform will return directly. Otherwise, perform will suspend the current fiber and arrange to continue only when the operation completes.

The wrap method "decorates" an operation, returning a new operation that, if and when it completes, will "wrap" the result of the completed operation with a function, by applying the function to the result. It's useful to distinguish the sub-operations of a non-deterministic choice from each other.

Here our new_op function will take an array of operation implementations and return an operation that, when performed, will synchronize on the first available operation. As you can see, it already has the equivalent of Go's select built in.

function new_op(impls)
   local op = { impls=impls }
   
   function op.perform()
      for _,impl in ipairs(impls) do
         local success, val = impl.try()
         if success then return impl.wrap(val) end
      end
      local function block(fiber)
         local suspension = new_suspension(fiber)
         for _,impl in ipairs(impls) do
            impl.block(suspension, impl.wrap)
         end
      end
      local wrap, val = suspend_current_fiber(block)
      return wrap(val)
   end

   function op.wrap(f)
      local wrapped = {}
      for _, impl in ipairs(impls) do
         local function wrap(val)
            return f(impl.wrap(val))
         end
         local impl = new_op_impl(impl.try, impl.block, wrap)
         table.insert(wrapped, impl)
      end
      return new_op(wrapped)
   end

   return op
end

There's only one thing missing there, which is new_suspension. When you go to suspend a fiber because none of the operations that it's trying to do can complete directly (i.e. all of the try functions of its impls returned false), at that point the corresponding block functions will publish the fact that the fiber is waiting. However the fiber only waits until the first operation is ready; subsequent operations becoming ready should be ignored. The suspension is the object that manages this state.

function new_suspension(fiber)
   local waiting = true
   local suspension = {}
   function suspension.waiting() return waiting end
   function suspension.complete(wrap, val)
      assert(waiting)
      waiting = false
      local function resume()
         resume_fiber(fiber, wrap, val)
      end
      schedule_task(resume)
   end
   return suspension
end

As you can see, the suspension's complete method is also the bit that actually arranges to resume a suspended fiber.

Finally, just to round out the implementation, here's a function implementing non-deterministic choice from among a number of sub-operations:

function choice(...)
   local impls = {}
   for _, op in ipairs({...}) do
      for _, impl in ipairs(op.impls) do
         table.insert(impls, impl)
      end
   end
   return new_op(impls)
end

on cml

OK, I'm sure this seems a bit abstract at this point. Let's implement something concrete in terms of these primitives: channels.

Channels expose two similar but different kinds of operations: put operations, which try to send a value, and get operations, which try to receive a value. If there's a sender already waiting to send when we go to perform a get_op, the operation continues directly, and we resume the sender; otherwise the receiver publishes its suspension to a queue. The put_op case is similar.

Finally we add some synchronous put and get convenience methods, in terms of their corresponding CML operations.

function new_channel()
   local ch = {}
   -- Queues of suspended fibers waiting to get or put values
   -- via this channel.
   local getq, putq = {}, {}

   local function default_wrap(val) return val end
   local function is_empty(q) return #q == 0 end
   local function peek_front(q) return q[1] end
   local function pop_front(q) return table.remove(q, 1) end
   local function push_back(q, x) q[#q+1] = x end

   -- Since a suspension could complete in multiple ways
   -- because of non-deterministic choice, it could be that
   -- suspensions on a channel's putq or getq are already
   -- completed.  This helper removes already-completed
   -- suspensions.
   local function remove_stale_entries(q)
      local i = 1
      while i <= #q do
         if q[i].suspension.waiting() then
            i = i + 1
         else
            table.remove(q, i)
         end
      end
   end

   -- Make an operation that if and when it completes will
   -- rendezvous with a receiver fiber to send VAL over the
   -- channel.  Result of performing operation is nil.
   function ch.put_op(val)
      local function try()
         remove_stale_entries(getq)
         if is_empty(getq) then
            return false, nil
         else
            local remote = pop_front(getq)
            remote.suspension.complete(remote.wrap, val)
            return true, nil
         end
      end
      local function block(suspension, wrap)
         remove_stale_entries(putq)
         push_back(putq, {suspension=suspension, wrap=wrap, val=val})
      end
      return new_op({new_op_impl(try, block, default_wrap)})
   end

   -- Make an operation that if and when it completes will
   -- rendezvous with a sender fiber to receive one value from
   -- the channel.  Result is the value received.
   function ch.get_op()
      local function try()
         remove_stale_entries(putq)
         if is_empty(putq) then
            return false, nil
         else
            local remote = pop_front(putq)
            remote.suspension.complete(remote.wrap)
            return true, remote.val
         end
      end
      local function block(suspension, wrap)
         remove_stale_entries(getq)
         push_back(getq, {suspension=suspension, wrap=wrap})
      end
      return new_op({new_op_impl(try, block, default_wrap)})
   end

   function ch.put(val) return ch.put_op(val).perform() end
   function ch.get()    return ch.get_op().perform()    end

   return ch
end

a wee example

You might be wondering what it's like to program with channels in Lua, so here's a little example that shows a prime sieve based on channels. It's not a great example of concurrency in that it's not an inherently concurrent problem, but it's cute to show computations in terms of infinite streams.

function prime_sieve(count)
   local function sieve(p, rx)
      local tx = new_channel()
      spawn_fiber(function ()
         while true do
            local n = rx.get()
            if n % p ~= 0 then tx.put(n) end
         end
      end)
      return tx
   end

   local function integers_from(n)
      local tx = new_channel()
      spawn_fiber(function ()
         while true do
            tx.put(n)
            n = n + 1
         end
      end)
      return tx
   end

   local function primes()
      local tx = new_channel()
      spawn_fiber(function ()
         local rx = integers_from(2)
         while true do
            local p = rx.get()
            tx.put(p)
            rx = sieve(p, rx)
         end
      end)
      return tx
   end

   local done = false
   spawn_fiber(function()
      local rx = primes()
      for i=1,count do print(rx.get()) end
      done = true
   end)

   while not done do run_tasks() end
end

Here you also see an example of running the scheduler in the last line.

where next?

Let's put this into perspective: in a couple hundred lines of code, we've gone from minimal Lua to a language with lightweight multitasking, extensible CML-based operations, and CSP-style channels; truly a delight.

There are a number of possible ways to extend this code. One of them is to implement true multithreading, if the language you are working in supports that. In that case there are some small protocol modifications to take into account; see the notes on the Guile CML implementation and especially the Manticore Parallel CML project.

The implementation above is pleasantly small, but it could be faster with the choice of more specialized data structures. I think interested readers probably see a number of opportunities there.

In a library, you might want to avoid the global task_queue and implement nested or multiple independent schedulers, and of course in a parallel situation you'll want core-local schedulers as well.

The implementation above has no notion of time. What we did in the Snabb implementation of fibers was to implement a timer wheel, inspired by Juho Snellman's Ratas, and then add that timer wheel as a task source to Snabb's scheduler. In Snabb, every time the equivalent of run_tasks() is called, a scheduler asks its sources to schedule additional tasks. The timer wheel implementation schedules expired timers. It's straightforward to build CML timeout operations in terms of timers.

Additionally, your system probably has other external sources of communication, such as sockets. The trick to integrating sockets into fibers is to suspend the current fiber whenever an operation on a file descriptor would block, and arrange to resume it when the operation can proceed. Here's the implementation in Snabb.

The only difficult bit with getting nice nonblocking socket support is that you need to be able to suspend the calling thread when you see the EWOULDBLOCK condition, and for coroutines that is often only possible if you implemented the buffered I/O yourself. In Snabb that's what we did: we implemented a compatible replacement for Lua's built-in streams, in Lua. That lets us handle EWOULDBLOCK conditions in a flexible manner. Integrating epoll as a task source also lets us sleep when there are no runnable tasks.

Likewise in the Snabb context, we are also working on a TCP implementation. In that case you want to structure TCP endpoints as fibers, and arrange to suspend and resume them as appropriate, while also allowing timeouts. I think the scheduler and CML patterns are going to allow us to do that without much trouble. (Of course, the TCP implementation will give us lots of trouble!)

Additionally your system might want to communicate with fibers from other threads. It's entirely possible to implement CML on top of pthreads, and it's entirely possible as well to support communication between pthreads and fibers. If this is interesting to you, see Guile's implementation.

When I talked about fibers in an earlier article, I built them in terms of delimited continuations. Delimited continuations are fun and more expressive than coroutines, but it turns out that for fibers, all you need is the expressive power of coroutines -- multi-shot continuations aren't useful. Also I think the presentation might be more straightforward. So if all your language has is coroutines, that's still good enough.

There are many more kinds of standard CML operations; implementing those is also another next step. In particular, I have found semaphores and condition variables to be quite useful. Also, standard CML supports "guards", invoked when an operation is performed, and "nacks", invoked when an operation is definitively not performed because a choice selected some other operation. These can be layered on top; see the Parallel CML paper for notes on "primitive CML".

Also, the choice operator above is left-biased: it will prefer earlier impls over later ones. You might want to not always start with the first impl in the list.

The scheduler shown above is the simplest thing I could come up with. You may want to experiment with other scheduling algorithms, e.g. capability-based scheduling, or kill-safe abstractions. Do it!

Or, it could be you already have a scheduler, like some kind of main loop that's already there. Cool, you can use it directly -- all that fibers needs is some way to schedule functions to run.

godspeed

In summary, I think Concurrent ML should be better-known. Its simplicity and expressivity make it a valuable part of any concurrent system. Already in Snabb it helped us solve some longstanding gnarly issues by making the right solutions expressible.

As Adam Solove says, Concurrent ML is great, but it has a branding problem. Its ideas haven't penetrated the industrial concurrent programming world to the extent that they should. This article is another attempt to try to get the word out. Thanks to Adam for the observation that CML is really a protocol; I'm sure the concepts could be made even more clear, but at least this is a step forward.

All the code in this article is up on a gitlab snippet along with instructions for running the example program from the command line. Give it a go, and happy hacking with CML!

Alberto Ruiz: GNOME Performance Hackfest

Eisha Chen-yen-su: Internationalization of Fractal (part 2)

A textual program using gettext

For my investigations, I first tried to write a textual program that works with gettext. I spent quite some time to figure out how all of this works but I finally was able to make it work. And that means that we should be able to implement i18n for Fractal using gettext!

I created a new Rust binary project with cargo and I added gettext-rs as a dependency in the Cargo.toml file by adding this line:

gettext-rs = { git = "https://github.com/Koka/gettext-rs", features = ["gettext-system"] }

I edited the main.rs source file to have a basic test to see if gettext-rs works correctly, here is its content:

extern crate gettextrs;
use gettextrs::*;

fn main() {
     setlocale(LocaleCategory::LcAll, "");
     bindtextdomain("test", "./po");
     textdomain("test");

    println!("Original: Open");
    println!("Translated: {}", gettext("Open"));

    println!("Original: Text Editor");
    println!("Translated singular: {}", ngettext("Text Editor", "Text Editors", 1));
    println!("Translated plural: {}", ngettext("Text Editor", "Text Editors", 2));
}

In the root directory of the project, I created a directory named “po”, then a subdirectory named “fr” (as I want to do a little French translation) inside “po” and then again, a directory “LC_MESSAGES” inside “fr”. So that I had the same folder hierarchy inside my “po” directory as in “/usr/share/locale”, because gettext will look for the translations in MO files that are in directories like “[TARGET]/[LANGUAGE]/LC_MESSAGES/[PACKAGE].mo”. [TARGET] and [PACKAGE] are defined by the call “bindtextdomain([PACKAGE], [TARGET]);” in the program. So here, the package name is “test” and the targeted directory is “./po”.

Next, I used the utility xgettext to extract all the translatable strings from the source file with this command:

xgettext -o po/template.pot src/*

And I edited template.pot to replace “CHARSET” with “UTF-8”. To get the file in which I would do the French translation, I used:

msginit --locale=fr --input=po/template.pot --output=po/fr/test.po

Finally, I added the translations of the strings on the lines with “msgid” in the lines with “msgstr” and my PO file had this content:

# French translations for PACKAGE package.
# Copyright (C) 2018 THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# Eisha <eisha@ernesto>, 2018.
#
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2018-05-15 12:33+0200\n"
"PO-Revision-Date: 2018-05-15 12:34+0200\n"
"Last-Translator: Eisha <eisha@ernesto>\n"
"Language-Team: French\n"
"Language: fr\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"

#: ../src/main.rs:10
msgid "Open"
msgstr "Ouvrir"

#: ../src/main.rs:13 ../src/main.rs:14
msgid "Text Editor"
msgid_plural "Text Editors"
msgstr[0] "Un éditeur de texte"
msgstr[1] "Des éditeurs de texte"

I compiled the MO file like this:

mgsfmt po/fr/test.po  -o po/fr/LC_MESSAGES/test.mo

The the program was finally ready to test, so when launching:

LANG=en_US cargo run

I get this output:

Original: Open
Translated: Open
Original: Text Editor
Translated singular: Text Editor
Translated plural: Text Editors

And when I launch it like this:

LANG=fr_FR cargo run

I get the expected output:

Original: Open
Translated: Ouvrir
Original: Text Editor
Translated singular: Un éditeur de texte
Translated plural: Des éditeurs de texte

You can find the program I have written in this repository.

A GTK program using gettext

When I got this textual program working, I had to test it with a GTK program in order to know if gettext-rs could still translate the strings in the source file and most importantly, if/how it could also translate strings from Glade files.

So I wrote a GTK program (with the crate gtk-rs) for which I created an empty window with the following title “A window title from Glade” with Glade, I saved this minimal UI in the file ui/main_window.glade. And I wrote the following code in main.rs:

extern crate gtk;
extern crate gettextrs;

use gtk::prelude::*;

use gettextrs::*;

fn main() {
    setlocale(LocaleCategory::LcAll, "");
    bindtextdomain("test", "./po");
    textdomain("test");

    gtk::init().unwrap();

    let window: gtk::Window = gtk::Builder::new_from_file("./ui/main_window.glade")
        .get_object("main_window").expect("Failed to load the main window");

    // UI initialization
    let label = gtk::Label::new(gettext("A label from the source code").as_str());
    window.add(&label);
    window.show_all();

    window.connect_delete_event(|_, _| {
        gtk::main_quit();
        Inhibit(false)
    });

    gtk::main();
}

Then, I followed the same procedure as before to generate the POT, PO and MO files, except that I also indicated to xgettext to parse the Glade file like this:

xgettext -o po/template.pot src/* ui/*

So that the content of po/fr/test.po was (at the end of the procedure):

# French translations for PACKAGE package.
# Copyright (C) 2018 THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# Eisha <eisha@ernesto>, 2018.
#
msgid ""
msgstr ""
"Project-Id-Version: PACKAGE VERSION\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2018-05-15 19:16+0200\n"
"PO-Revision-Date: 2018-05-15 19:17+0200\n"
"Last-Translator: Eisha <eisha@ernesto>\n"
"Language-Team: French\n"
"Language: fr\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n > 1);\n"

#: ../src/main.rs:19
msgid "A label from the source code"
msgstr "Une étiquette du code source"

#: ../ui/main_window.glade:7
msgid "A window title from Glade"
msgstr "Un titre de fenêtre de Glade"

When running this program with:

LANG=en_US cargo run

The window opened is like this:

Capture du 2018-05-16 11-09-51

And when running it with:

LANG=fr_FR cargo run

The window opened is like this:

Capture du 2018-05-16 11-11-54

So it seems that GtkBuilder is calling gettext when parsing the UI file.

You can find this other program I have written in this repository.

So now I can start to implement i18n support within Fractal! Although I will also have to figure out how to properly integrate this with meson.

Alexander Larsson: Introducing the 1.8 freedesktop runtime in the gnome nightly builds

Tobias Bernard: Fractal Hackfest in Strasbourg

Last week we had an intense 4-day hackfest in Strasbourg to map out the future of Fractal, a native GNOME Matrix messaging app. The event was held at Epitech in Strasbourg’s old town, and organized by Alexandre Franke. Among the attendees were core Fractal contributors Daniel, Alexandre, Eisha, and Julian, as well as Dorota, Adrien, and Francois from Purism. Special thanks go to Matthew from the Matrix core team for joining us on the first two days.

Our main priorities for the hackfest were to plan the roadmap for the next months, decide on the tasks for our GSoC students (Eisha and Julian), and work on the design of some important missing features, like the room settings.

I personally attended the hackfest in both my role as designer on the Fractal project and as a Purism employee currently working on the apps for the Librem 5. One of the reasons why several members of the Librem 5 team attended the hackfest was that we will need a Matrix messaging app on the phone and wanted to explore a potential collaboration.

The hackfest was extremely productive, so much so that I’ll need multiple blog posts to report on all the things we worked on. Here’s a quick outline of some of the most important things that happened:

  • We’re splitting the app into two separate apps (more on this in a future blog post)
  • A big refactor of the backend is happening soon to enable the split
  • We discussed having a system-level Matrix daemon, which different apps could use as a backend (e.g. the two different messaging apps and a calls app)
  • Matthew explained that room types will be simplified into 1-1, private groups and public groups in the future (which nicely complements our split). We discussed whether 1-1s should be immutable (they should :P)
  • Matthew explained how end-to-end encryption and calls work in Matrix, and how we could get them in Fractal
  • We came up with an initial design for multi-account (which basically consists of an account chooser at startup, and a separate window for each account)
  • We discussed a design for read receipts. Not quite done yet, but we’re on the right path, I think.
  • We talked about what it will take to make Fractal work on mobile. Not too big of a problem design-wise, but we’ll need Rust bindings for libhandy and emeus
  • Eisha will be investigating i18n, because we really want to make the app translatable (currently this is hard to do because we use Rust)
  • Julian will be working on a big message history UI refactor/redesign, as well as other UI stuff, such as user account settings
  •  There is a huge number of message types we don’t support yet and we discussed the design for most of them (including in-app viewers and a history of sent files)
  • With Matthew’s help, Daniel fixed the slow initial sync
  • Julian worked on a nicer UI for the GTK emoji chooser and Rust bindings for gspell
  • Alexandre landed the new, prettier default avatars
  • Quentin started packaging Fractal for macOS
  • We fought our way though the garbage fire that is the Join and History settings in Riot, and emerged with a design that isn’t terrible (thanks Dorota and Julian!)
Matthew explaining device verification for E2E chats

Thanks everyone for attending, Epitech and Alexandre for the venue, and Purism for sponsoring the hackfest! It was amazingly productive to have everyone in one place, and I look forward to seeing you all at GUADEC in July :)

 

Jehan Pagès: GIMP 2.10.0 is out!
Older posts are this way If this message doesn't go away, click anywhere on the page to continue loading posts.
Could not load more posts
Maybe Soup is currently being updated? I'll try again automatically in a few seconds...
Just a second, loading more posts...
You've reached the end.

Don't be the product, buy the product!

Schweinderl