Category Archives: programming

Meta-category for programming posts that are not related directly to a project

Playing with a native C extension for Node.js

  By |
Leave a comment

I’ve always felt like a good way to fully understand a language is to play with the native interface it exposes. I’ve already done a post playing with a native C Ruby extension, and now that I’m using Node.js more I felt like I should dig into it. Here is how I ended up playing with a native C extension for Node.js. I did not have a real goal while playing around, so I did the exact same thing I did for Ruby. As always, the code is hosted on my GitHub repository!

Why a native extension

Node.js is a great tool for event-based concurrency. Having a garbage collector and being single-threaded makes for a language that is easy to get into and fast. But like any language, it is not perfect.

For example, Node.js is not the best if you need to do some CPU-bound computations. Even using threads won’t help since only one Node.js thread will run at a time. There is also the case of wanting to interface with existing native libraries or with the OS.

For all these reasons, and many others, a native extension could be useful.

Needed setup

There are multiple tooling frameworks that can be used to do a native library for Node.js. In the end, most end up wrapping around N-API, the Node.js native API. I decided to use node-gyp.

The initial repository setup was quite trivial:

  1. Create (and fill) the binding.gyp file
  2. Run npm init

The binding.gyp file is used to configure how node-gyp will build the native part of the extension. In my case, I went with something simple:

Once you have the file in your directory, npm init will recognize it and do the rest of the setup.

Loading the native interface can be tricky, luckily there is a package for that! The library is called node-bindings. Using this library, you can load your library with one simple line: require('bindings')('nci.node').

This will find your native bundle (in my case nci.node) and just load it. No need to think about release vs debug, about platforms, or anything else.

Module management

Exposing your module is done through the  NAPI_MODULE macro. The macro requires a function as a second argument, this function will be called to initialize the module. The initialization method needs to return everything that is exported from the module. For example, the following would register a new module with the method Init as initializer: NAPI_MODULE(NODE_GYP_MODULE_NAME, Init).

In my case, I add the newly created class definition to the existing exports and return that. I also take this time to store a few global variables that I’ll need to reuse in various calls.

Creating the class definition is simple enough: register the class by giving it a name and a constructor callback and then export it. For example, I went with the following:

Class/instance management

There are two Node.js syntaxes that can get your constructor called:

I decided to support only the first version simply because it is a bit easier to do so. In order to distinguish the two versions, you’ll need to check the target of the call. Basically, if the target is set, you are in the  new MyClass()  version. Doing so is pretty easy:

Once in the constructor, I get and validate the first argument, expecting an int. Once this is done, I simply create the native instance and wrap it like so:

When wrapping the instance is the moment where you define a cleanup callback. That callback will be called when the object is garbage collected.

Once the object is wrapped, you’ll need to register all functions on the instance. In order to do so, simply repeatedly call napi_create_function and napi_set_named_property to create and add the function to the object.

Accessing the native object

In the function callbacks, you’ll need to access your native object. In order to do so, you’ll need to get the call information and then unwrap the target object (this) like so:

Loading the module

If you are using node-bindings like me, loading the library is done simply through:  require('bindings')('nci.node'). It is a good idea to wrap that into your own module and not require your customer to do this for you.

Related links

A tale of C++ native Ruby and RAII

  By |
Leave a comment

While I was playing with my last project, I wanted to gather the execution time of a few native functions. Doing so in C is a little bit painful: it requires quite a bit of code, temporary variable, and such. One really powerful idiom I liked using while doing C++ was Resource Acquisition Is Initialization (RAII) to do this kind of task. This post will contain two subjects: using RAII to time function execution and using rake-compiler with C++ for native Ruby. So here is a tale of C++ native Ruby and RAII.

What is RAII

In a class, RAII is represented in a really simple way: the constructor acquires something (file handle, lock, etc.) and the destructor releases it.

An example of this is the std::ofstream class. An instance of this class acquires a handle on the given file and the destructor releases it. Therefore, the following is completely valid:

This idiom ensures that the underlying resource gets released correctly in all cases. No need to catch exceptions and rethrow in order to manually close the file, it will be done when the stack unwinds.

Using RAII to time function execution

In order to use RAII to time function execution, I went with a really simple flow:

  • The constructor gets the start time
  • The destructor gets the end time and prints the elapsed time

The code using the previous flow can be found here:

Using this class becomes super simple: create an instance and let it go out of scope. An example of this can be found in this file:

Using rake-compiler with C++

I was expecting some major differences when using C++ instead of C with rake-compiler. Turns out that the tool does most of the heavy lifting. I only needed to put the C++ files in the directory and they got built magically.

There were only a few things that I needed to care about:

  • Ensure that the entrypoints were in an Extern C section in order for the method signatures to be valid
  • Define a typedef to enforce C signatures whenever you call a C Ruby function that requires a function pointer
  • In order to manage memory correctly, remember to delete any allocated instances

Using rake-compiler to build native Ruby extensions

  By |
Leave a comment

Having renewed my love for my Raspberry Pi, I wanted to make a quick library to access a temperature sensor through Ruby. This sensor uses a single-bus format, and one communication lasts about 4ms. I felt like this was good enough of an excuse to play with the native layer of Ruby. Here is how I ended up using rake-compiler to build native Ruby extensions.

Quick disclaimer: most of what I did was heavily inspired by a RubyGems guide. I’ll try to point out things that I had most issues with or that I found particularly interesting. All the code associated with this post is hosted on my GitHub project.

Adding a dependency on an external library

Adding a dependency on an external library was one of the things that were not completely clear. In my case, I needed to use libgpiod. In order to do so, I had to modify extconf.rb to add the following lines:

These two lines ensure that the generated Makefile will include the right search directories for the headers and link to the right library.

Controlling the native classes’ modules

I ended up with a really simple solution for handling my modules: the module is created in Ruby, and then fetched in C and used there. In the sample project, the module creation is done in the main Ruby file and then used when initializing the C module.

Getting the module was done easily: rb_const_get(rb_cObject, rb_intern("NCI")). Once this was done, creating the class was done using the handy rb_define_class_under:  rb_define_class_under(mNCI, "NCINativeDevice", rb_cObject). The third parameter to this last function is the superclass, in this case the Object class.

Associating a C structure with the Ruby instance

Three functions come into play when using a C structure: nci_native_device_alloc, nci_native_device_init, and nci_native_device_free.

The first function (_alloc) is used to allocate an empty C structure. This function is not the initializer, it will not get any of the arguments passed on initialization. The important part of the function is the call to Data_Make_Struct. This function will allocate the given C structure, but won’t initialize it for you.

The second function (_init) is the true initializer. It will receive the arguments that are passed in the Ruby code. In order to store these values in the C structure, you need to extract the C pointer from the instance. This is done using Data_Get_Struct. Weirdly enough, this won’t return the pointer to the C structure, it stores it in the last argument.

The last function (_free) is the most important of all. It is responsible for deallocating whatever you allocated. The C structure (allocated for you by Data_Make_Struct) does not need to be deallocated, the system will do it for you.

Various sources

Custom initializer when subclassing Dry::Struct

  By |
Leave a comment

I’ve been using dry-struct in most of my Ruby projects. This time around, I had to do something a little bit unusual: I have a subclass that requires less information than its parent. So here is how I wrote my custom initializer when subclassing Dry::Struct.

The Problem

The solution seems simple: the subclass should simply not send part of the information to its base constructor. But Dry::Struct was built in a way that we can’t do that as easily. The first thing I tried was the following:

Line 21 works correctly, but this is not what I wanted to achieve. I wanted to not have to specify X when creating the object, like the line 22. When doing so, I got greeted with a: in `block in resolve_missing_keys': :X is missing in Hash input (Dry::Types::MissingKeyError).

After investigating the callstack, I figured that the validation was not done in the initializer, but in the class’s new function.

The Solution

Armed with the information that the validation is done in the new function, I tried something else. The following is the version I ended up with:

The magic is on line 15: instead of defining an initializer, I overload the new function of my class. This allows me the same control as an initializer but does this before Dry::Struct validations.

Now, is this the best solution … probably not. But following the documentation, I could not find any other ways to do it.

Using SemanticLogger with Highline in Ruby

  By |
Leave a comment

I started using Ruby again for a command-line project of mine. In order to do all that I needed to do, I had to find a configurable library for logging (kinda like log4net and friends) and another one to ask various input from the command line. Therefore, I ended up using SemanticLogger with Highline. I ended up having a few issues, here is how I fixed them.

The problem

The first input I did, after showing a few debug information, ended up a mangled mess looking like:

You can clearly see that the output is intertwined with the input selection. This is really annoying and needed fixing.

The associated code did not look anything weird, even a simple test like the following produces the issue:

The only explanation for me was that the output from the logger was done in an asynchronous way, either some kind of buffering or simply done in a background thread. After checking a bit more  Highline’s homepage, I found the answer:

Logging is performed in a separate thread so as not to slow down the application whilst logging to one or more destinations.

The solution

After looking through more guides from the official site, I finally found what I needed: a way to make SemanticLogger log in the current thread.

There are multiple ways of enabling this feature, I decided to go with the code-based approach: add  ::SemanticLogger.sync! before any of my appenders got created.

Doing so, my final solution looks like:

And my output is finally as expected:

Using HK2 in a basic Jersey Web Service

  By |
Leave a comment

At work, I do a lot of backend work in C#. I’ve always felt that seeing other frameworks are good to get better. This time, I decided to go with Java, more precisely Jersey.  I’ve come to love dependency injection in .NetCore, therefore using HK2 in a basic Jersey Web Service was first on my play list.

Project bootstrap

I decided to go for a really simple project and bootstrap it using Maven:

This generates a simple server with a single HTTP GET endpoint.

Playing around

I then started adding a few controllers and workflows. Using this setup, each controller needs a handle to a workflow. Of course, this is easier done using dependency injection. I did not want to reuse what I did for my other project, so I explored what Jersey proposes. Quickly I found that Jersey comes with HK2 all ready, so I glimpsed the HK2 introduction and started playing with it.

I added a few @Contract , @Service  and @Inject , and tried it out. Before even waiting for the results, I knew that something was missing. It was either major magic underneath or I was not defining which service I wanted to use for each contract.

And as expected, I got a nice little exception:  org.glassfish.hk2.api.UnsatisfiedDependencyException: There was no object available for injection [snip]

Fixing it

After a tiny bit of googling and better reading on my part, I figured the issue: you simply need to register a binder into your server configuration. More explicitly, you’ll need the following:

Final part is to use this configuration class when initializing your server, like so:

Sources

  • Jersey, RESTful Web Services in Java
  • HK2, A light-weight and dynamic dependency injection framework

Why I’m not using Dagger2 on my project

  By |
Leave a comment

In order to make a Java side project of mine easier to evolve and test, I wanted to add dependency injection. After a quick lookup, I figured that Dagger2 would be a really nice little thing to add to my project. So here is why I’m not using Dagger2 on my project.

Disclaimer: this is only my opinion, Dagger2 seems really good and I’ll most probably use it in a future project where it is more suitable.

Context

My project is split into a core library (to be reused later) and the main project. Quite a bit of the library is built to be sub-classed while enforcing a certain flow, validation and features for the main project.

Experimentations

While tinkering with Dagger2, I did notice a few things that were annoying to me:

  • No support for inheritance;
  • Complex integration with Eclipse (through Gradle);
  • Lack of simple examples

The lack of support for inheritance was already a blocker for me. Still, in order to get a good feel of the library, I decided to try to get around the limitation. Doing so required me to manually inject into my super-class when building my instances. This exposed quite a bit of things that had no right being exposed to the subclass. It also made the code harder to evolve since I would need to update my sub-classes every time I add a dependency into my core library. That would require an API break where there should be none.

I’m new to the whole Gradle ecosystem, and getting it to work correctly with Eclipse was already quite a task (see this post and that post). Integrating Dagger2 into all of that proved to be really painful. I probably missed something somewhere, could simply be missing a bit of documentation somewhere. I do plan on trying to make it work again in the future and document how to get it all running.

Finally, Dagger2 allows support of really complex scenarios, but, sadly, it exposes a lot of complexity when trying to do something simple. This could be due to a lack of simple examples or simply a misunderstanding on my part. One way or another, I do plan on getting back to this to figure out how to use it in simple scenarios.

Solution

In the end, I decided to go with something extremely simple that supports only what I need:

  • Global Singletons;
  • Introspection is used only to inject the dependency manager in the constructor (if needed)

In the end, usage looks roughly like:

I did add a few other helpers in there. Nothing too fancy, just quality of life.

Interacting with Google Tasks on Android

  By |
Leave a comment

Since I’ve been playing with Google Tasks, I wanted to look at my task from my Android phone. Sadly, all the application I could find were using a lot more permissions than I like to give. Therefore, I decided to look into why all these permissions are needed. This will be a little tutorial on interacting with Google Tasks on Android.

Disclaimer: I have not yet published an application using this, so it is possible that something is missing. I will update this post as soon as I have more information (or remove this disclaimer if nothing more is needed).

Project Setup

There are two basic files that need modification in order to be able to use Google Tasks on Android.

The first files you need to modify is the AndroidManifest.xml file. You will need to add the android.permission.INTERNET by adding the following line:

This permission is used to allow your application to connect to the Internet.

The second file that needs modification is the app/build.gradle file. Adding the following lines to this file will add the needed libraries to your project’s dependencies:

Do note that I had to force the version of com.google.code.findbugs:jsr305 in order to fix a dependency issue.

Google API Setup

Configuring the Google Backend services for Android is a little bit different from doing so for desktop applications. I already did a quick tutorial on doing the configuration for Android here.

Handling the user’s credentials

In order to get the user’s credentials, you will need to ask for his permission. I went the easy route and asked directly for read/write access to Google Tasks, but you can get the list of scopes that can be used here. In order to be able to get the user’s token, you need to add the requestEmail permission.

Basically, the way the permission flow works is:

  1. Build a sign-in intent (with the desired permissions);
  2. Ask the system to fulfill the intent and wait for the response;
  3. Process the response

The code is quite straightforward:

Initializing the Google Tasks API

In order to initialize the Google Tasks API, you will need 3 things:

  • A JSONFactory
    I went with an instance of com.google.api.client.json.jackson2.JacksonFactory simply because it was there and worked
  • A HttpTransport
    Depending on the Android version you are targeting, you must use a different class. Using AndroidHttp.newCompatibleTransport() makes this selection transparent.
  • The user’s credentials

Interacting with Google Tasks on Android

Once you got the Tasks Service in your hands, interacting with it is the same as doing so in Java. Since I’ve been using API 23, I can’t use streams like I used in my java example. Therefore, I did two little helpers ( TaskList findTaskList(TaskLists tasklists, String title)  and Task findTask(Tasks tasks, String title)).

Creating a task list

Finding a task list

Creating a task

Finding a task

Updating a task

Completing a task

Handling a large number of return values

Like any good online service, Google Tasks API handles a large number of values using pagination. By default, most APIs will return a maximum of 100 results. You are allowed to change this value, but in order to get good performance, you should not put it bigger.

Basically, the way to handle pagination is: do your normal call, check if you got a page token and if you do you know there are more results waiting. You can then call the API again with the page token in order to get the next result set:

Sources

Interacting with Google Tasks using Java

  By |
Leave a comment

I already did a post giving an overview of interacting with Google Tasks using Ruby, but with the change in scope of my project, I now need to do the same in Java. Therefore, here is an overview of interacting with Google Tasks using Java. Do note that this probably won’t work on Android, it is meant to be used from Java Desktop editions.

Project Setup

Using gradle, the project setup is quite simple:

Google API Setup

In order to access Google Tasks using Java, you will need to use Google’s OAuth. Google’s OAuth uses what they call client secrets file. These files contain all the information needed to access the selected services.

To get a client secrets file, you can follow my other post.

Handling the user’s credentials

This project is my first Java project since a long time ago. I quickly remembered how painful it can be to play with the interface/abstract class hell. You have to instantiate 20 different things before calling anything and selecting the right implementation, or even simply finding an implementation, is sometimes tricky. Here, I selected the following implementations:

  • JsonFactory: com.google.api.client.json.jackson2.JacksonFactory, it was there and worked, so I used it;
  • HttpTransport: com.google.api.client.http.javanet.NetHttpTransport, according to the API’s documentation “NetHttpTransport is based on the HttpURLConnection built into the Java SDK, so it is normally the preferred choice“;
  • DataStoreFactory: com.google.api.client.util.store.FileDataStoreFactory, this implementation will store the tokens in a file (Ok … multiple files …). In my case, it will be in a directory called tokens in the current working directory;

Here is the authentication code I used. It will take care of caching the tokens, opening the browser if a new token is needed and getting the result token.

Initializing the Google Tasks API

In order to use the Google Tasks API, you need to build the service. Doing so requires most of the things that were initialized in the previous section:

Interacting with Google Tasks using Java

Creating a task list

Once you have all the boilerplate done for the initialization of the API and the service, creating a tasklist becomes quite easy:

Finding a task list

The Google Tasks API only allows you to search by tasklist ID, not a title. This ID is automatically generated and therefore not really easy to use. The easiest way to find a task list is to list all task lists and find the one you are looking for in there:

Creating a task

In order to create a task, you first need a tasklist. Once you got that, creating the task is easy enough:

Finding a task

Just like finding a tasklist, you can’t directly search by task title. Therefore, you need to go iterate over the full list to find the one you are looking for:

Updating a task

Once you have a task (and a tasklist) in your hands, updating most of its fields can be done directly on the Task instance. The following example updates the due date of a task:

Completing a task

Just like updating a task, completing it is quite easy: set its status field to completed:

Handling a large number of return values

Like any good online service, Google Tasks API handles a large number of values using pagination. By default, most APIs will return a maximum of 100 results. You are allowed to change this value, but in order to get good performance, you should not put it bigger.

Basically, the way to handle pagination is: do your normal call, check if you got a page token and if you do you know there are more results waiting. You can then call the API again with the page token in order to get the next result set:

 

Sources

Interacting with Google Tasks using Ruby

  By |
Leave a comment

On the first iteration of a project of mine, I wanted to access Google Tasks using Ruby. Therefore, I did a little bit of tinkering with using the Google Tasks API using Ruby. In this post, I will go over your project’s setup, the Google backend setup and the basic usage of the API.

Project Setup

All of the Google Tasks APIs are basically REST endpoints. One could simply call all these HTTP endpoints manually, but this would be painful. I decided to use google-api-client to do all the heavy lifting. In order to include it in your project, simply add the following to your Gemfile:

Google API Setup

In order to access Google Tasks using Ruby, you will need to use Google’s OAuth. Google’s OAuth uses what they call client secrets file. These files contain all the information needed to access the selected services.

To get a client secrets file, you can follow my other post.

Handling the user’s credentials

The way the Google APIs work, an URL will be provided to you. Opening that URL and grabbing the authentication token is all on you. Therefore, there is three consideration when handling the user’s credential:

  1. Storing the credentials (so that you don’t ask access every time);
  2. Showing the consent webpage;
  3. Grabbing the authentication token;

For this quick test, I decided to use a file storage for the token. The FileTokenStore token store will simply use a YAML file to store the user’s tokens. To show the consent webpage, I went with launchy. This cross-platform library will handle opening URLs in the default browser and such. Finally, I decided to basically do nothing to grab the token: I expect the user to paste it on the command line.

Initializing the Google Tasks API

Now that you have the user’s credentials, you can initialize the API you want to use. In my case, this is the task API. In order to do so, simply create a new instance of the service you want to use (Google::Apis::TasksV1::TasksService) and set its authorization member to the credentials you got from the client.

Interacting with Google Tasks using Ruby

Creating a task list

The tasklist is basically a grouping of tasks. It is the root concept of Google Tasks. Creating one is quite straightforward:

Finding a task list

The Google Tasks API only allows you to search by tasklist ID, not a title. This ID is automatically generated and therefore not really easy to use. The easiest way to find a task list is to list all task lists and find the one you are looking for in there:

Creating a task

Creating a simple task, at the top of the task list is quite straightforward:

Finding a task

Just like finding a task list, finding a task by its title can’t be done directly through the API. In order to do so, you need to iterate over the tasks and check them:

Updating a task

Once you have your hand on the task you want to update, simply update the field you want to update and call update_task:

Completing a task

Completing a task is done by setting its status to “completed” and updating the task:

Handling a large number of return values

Google’s Task API uses pagination in order to handle a large number of return values. By default, a maximum of 100 return values will be returned by calls to list_tasklists and list_tasks. The return values of these two APIs will contain a next_page_token value. You can then use this token in a subsequent call to the API specifying the optional argument page_token in order to get another set of return values.

Here is a quick example of handling of pagination with the list_tasks call. Do note that in this example, I set the max_results value in the call so that I do not have to create 100+ tasks.

Sources