biogems.info

bio-hmmer_model 0.0.1 released

Fri, 18 May 2012 00:00:00 +0200

Parse PFAM HMM definition files

bio-hmmer3_report 0.0.1 released

Fri, 18 May 2012 00:00:00 +0200

Enables parsing of HMMER version 3 reports

bio-pileup_iterator 0.0.1 released

Thu, 17 May 2012 00:00:00 +0200

Iterate through a samtools pileup file

bio-gag 0.0.1 released

Thu, 17 May 2012 00:00:00 +0200

bio-gag is a biogem for detecting and correcting a particular type of error that occurs/occurred in particular versions of the IonTorrent DNA sequencing kit

GSoC weekly report #0.5 (Artem's blog)

Sun, 13 May 2012 21:55:29 +0200

So, another week of “pre-coding” period have passed. I was mainly concentrated on D part of the project, adding functionality to it. During the week, I implemented parsing of the whole BAM file :) Today I wrote a simple utility in D, which uses my library to convert BAM to SAM. It doesn’t work with [...]

GSoC weekly status report No.1.1 (Marjan's blog)

Sat, 12 May 2012 21:40:00 +0200

This year we the GSoC students sure are a very creative group, just look at our numbering schemes for our status reports for the pre-coding period - everyone has his own thing going :)

And now back to the GFF3 project. I found a few more sites with big GFF3 files, those will be great for performance testing. And Robert Buels suggested that I should reuse the test suite from the Perl’s Bio::GFF3::LowLevel::Parser, and I think that’s a great idea. I should definitely use that for completeness testing and I will check the test suites of other GFF3 parsers.

I have also finished the work for the first week. That means basically I’m already more then two weeks ahead of schedule. The parser is now reading data on the D side and forwarding that to Ruby line by line. That won’t be faster then reading the file from Ruby, but that’s a nice basic case to get data flowing from D to Ruby.

The rake tasks have been improved too. There are now two tasks for building the D library, “compile” and “compiledebug”, and there is the “spec” task for running rspec tests and “features” task for running cucumber tests. The “clean” task now deletes object and library files.

There is also a problem with the D library and garbage collector. It seems this is the problem Iain Buclaw (one of the GDC developers) has warned us about. When using a D shared library, when the GC kicks in for the first time, it looks like if it collects all the static data, for example the per-module variables. And pretty much everything, even when we register with GC a chuck of memory allocated with malloc, it still gets collected. Or at least that’s what it looks like. However, Iain also assured us that this will be solved by the end of this month/beginning of the next. My cucumber and rspec tests still work because they don’t require enough memory for the GC to run, but to be sure that this issue doesn’t interfere with development at this point, I manually disabled the GC on library initialization. I didn’t try yet, but from what has been discussed in the forums, both 32 and 64-bit DLLs on windows built using DMD work fine.

I also helped Pjotr with getting our blog posts included in the RSS feed on biogems.info.

bio-nexml 1.1.0 released

Wed, 09 May 2012 00:00:00 +0200

BioRuby plugin for reading and writing NeXML (http://nexml.org)

bio-tabix 0.1.3 released

Mon, 07 May 2012 00:00:00 +0200

Ruby binding for samtools tabix

[GSoC] Weekly report #0 (Artem's blog)

Sun, 06 May 2012 21:14:43 +0200

Finally, OBF seems to be convinced about usefulness of new BAM parser. Thus now I can spend time on coding instead of writing/talking on why what I’m about to do is so important :) I began to write code before the official start for two reasons. First, I just can’t wait to get my hands on [...]

bio-cnls_screenscraper 0.3.0 released

Sun, 06 May 2012 00:00:00 +0200

Programmatic interface to the cNLS nuclear localisation signal prediction software

GSoC weekly status report No.1 (Marjan's blog)

Fri, 04 May 2012 15:26:00 +0200

It has been 10 days since the GSoC results were published, and a lot has happened since then. I got to know the other students and mentors in a longish meeting on Google hangout, I got into a discussion with my mentor on IRC in which we didn’t agree about the parallelization strategy for the parser (experiments will show who’s right) and my inbox is full with mails from my mentor and other students, in which we exchanged loads of interesting ideas. Also, there was a bug in biogems.info site…

biogems.info

We had a bug in the biogems.info website last weekend. The part of the code that was retrieving commit history data from GitHub wasn’t resistant to fluctuations in github.com availability. It seems GitHub had some problems over the weekend (probably a lot of people were working on their hobby projects) and the script that retrieves data and generates the website was failing after it has retrieved data for only a couple of biogems. Making the script ask again for the same data in case the first request wasn’t successful solved the problem.

Here is the link to the pull-request with the fix:

https://github.com/pjotrp/biogems.info/pull/13

Administrative

There is now a Git repository on GitHub for my GSoC project:

https://github.com/mamarjan/bioruby-hpc-gff3

Coding

I have already started doing the work that was planned for the first week of the coding period, to be sure the Ruby interpreter will play along nicely with the D runtime. The first impressions are good and here are now some cucumber tests and features, documenting most of the plugin interface that was intended for the first week (iterating over lines) and there are a few rake tasks for building the D library and running tests. However, the D part for the first week is still not finished.

Currently only Ubuntu 32-bit and the DMD compiler are supported. This will of course be extended over the summer.

I’m also looking into extending the rake-compiler gem to support D. Currently it can be used to build native gems written in C or Java, and there are quite a few gems that are using it. This would make it possible for future gems using D to have a similar directory structure and to reuse the tasks for compiling D libraries on different platforms.

64-bit shared libraries

There is currently no way to build 64-bit shared libraries on Linux with any of the three available D compilers. And we can expect that most users are going to use 64-bit operating systems along with 64-bit versions of Ruby.

Because of that we contacted the author of the GDC compiler, which is open source and in a good shape, and received info that this problem is scheduled to be solved within a month or two. With that info, in a discussion with my Pjotr and Artem, we decided that it’s good enough for us to continue the parser projects in D. We can start working on our projects using 32-bit builds of Ruby, and once the support is available we can start testing our applications in 64-bit mode.

Also, in windows, there are no issues creating 64-bit DLLs, so that we don’t expect any problems with that platform.

The first performance test

At the moment there are two approaches in regard to the parallelization of the parser that I would like to compare and choose the better one. In one approach the Ruby code simply sends the file path to the D library and receives parsed data. In the other, Ruby code reads data from the file and passes it in blocks of data to the D library, which then distributes the work to multiple actors.

Just to be sure that reading the data from the file into Ruby strings won’t be that expensive, I made a quick performance test. Here is the script:

filename = ARGV[0]

open(filename, "r") do |f|
  while f.read(1024*1024)
  end
end

It reads the whole file a block at a time.

The results are encouraging. Once the 1.1GB GFF3 test file was cached in memory, the script needed only 0.6 seconds to finish reading that file again (on my Intel i3 laptop with 4GB laptop).

There is however one point I need to add. When the block size is reduced to 512 bytes, that is when only 512 bytes are read at a time, the script requires 4.5 seconds instead of 0.6. So, for best performance, it is important that the Ruby part of the parser reads data in big chunks, and send it to the D parsing code.

What’s next?

What follows is that I’m going to search for applications which are using GFF3 files and ask around for examples that might be in development and that would benefit from a fast sequential parser. Also, a list of GFF3 files from different sources would help me make the parser more robust and useful.

Also, there seems to be huge interest for a GFF3 parser with more features, like indexing, random access and writing output, and also support for linking into trees of features that are not located close to each other in the file. A fast sequential parser could be used to generate indexes, and the lower-level parts can be used to reorder the file for faster future usage. Based on that, I think this project is a good start.

So, again, I would like to ask you if you’re using the GFF3/GTF file formats in your research, to send me example files and descriptions of how are your applications using the data. This way I’ll be able to test the parser against your files and optimize it for your applications. Thank you! :)

Reading BAM files in D (Artem's blog)

Fri, 04 May 2012 11:19:46 +0200

It turned out to be really easy, using std.stream and std.zlib modules :) Sequence of actions is as follows: Create an instance of BufferedFile, providing file name to its constructor. Then pass it to constructor of EndianStream, specifying little endian byte order Read BGZF blocks from this stream, according to their specification, and decompress them [...]

bio-tm_hmm 0.2.1 released

Thu, 03 May 2012 00:00:00 +0200

A bioruby plugin for interaction with the transmembrane predictor TMHMM

On programming languages and HPC (Artem's blog)

Tue, 01 May 2012 10:55:45 +0200

Some people believe one should never rewrite old code. I’m not among them, and let me tell why, taking samtools as an example. If you want your programs to scale on modern multicores, you must use parallelism. Free lunch is over, damn it! It’s great that samtools’ authors realize that, but hey, do you really [...]

bio-rdf 0.0.1 released

Tue, 01 May 2012 00:00:00 +0200

Bioinformatics triple-store support

bio-faster 0.4.4 released

Fri, 27 Apr 2012 00:00:00 +0200

A fast parser for FastQ files

[GSoC] Intro (Artem's blog)

Thu, 26 Apr 2012 21:46:51 +0200

So… I’m accepted to GSoC this year, and that’s cool! What is more cool, though, is that we have a great team in BioRuby community, working on similar tasks, and everybody is interested in performance and robustness. For me, it’s quite challenging, because the other two guys have professional experience and… uhm, a bit older [...]

GSoC 2012 (Marjan's blog)

Wed, 25 Apr 2012 18:15:00 +0200

This will the first post in a series about my participation in GSoC this summer. I’m very happy that the community around OBF liked my proposal, and I’m also very thrilled to become part of the BioRuby community!

My project title is “The worlds fastest parallelized GFF3/GTF parser in D, and an interfacing biogem plugin for Ruby” and the abstract is located at the following location:

http://www.google-melange.com/gsoc/project/google/gsoc2012/zaki/28001

I’m looking forward to a great summer this year :)

Bug hunting (Artem's blog)

Sat, 21 Apr 2012 21:43:34 +0200

It might seem that everything is clear about D and Ruby interaction. Not quite so, as it turns out. Let’s consider this (almost) trivial example: test.d: test.rb: (Full code and Makefile is available at https://gist.github.com/2366623) Running this Ruby code leads to a segmentation fault with the following backtrace: What the hell is going on?! Let’s investigate… [...]

bio-gem 1.3.4 released

Wed, 18 Apr 2012 00:00:00 +0200

Biogem is a software generator for Ruby in bioinformatics

bio-samtools 0.5.3 released

Sat, 14 Apr 2012 00:00:00 +0200

Binder of samtools for ruby, on the top of FFI.

D & Ruby FFI, part 2: working with arrays and strings (Artem's blog)

Tue, 10 Apr 2012 20:22:39 +0200

Firstly, string in D is nothing but immutable(char)[], that is, an array of particular type. Secondly, the memory layout of an array is simple: size_t (for storing length) + pointer (which points to the array elements). The immediate conclusion is that D strings are not necessarily zero-terminated. It’s not very convenient when we’re working via FFI; the [...]

D & Ruby FFI, part 1: mapping D structs onto Ruby classes (Artem's blog)

Tue, 10 Apr 2012 14:42:12 +0200

This post starts series about interaction between D and Ruby via FFI. Several things are to be investigated. Firstly, both languages have their own garbage collectors. Although we can disable D garbage collector manually for some classes/structs, that’s not convenient at all. We want to use existing D code without modifying it. Also, D ABI is [...]

bio-signalp 0.2.3 released

Wed, 04 Apr 2012 00:00:00 +0200

A wrapper for the signal peptide prediction algorith SignalP

bio-logger 1.0.1 released

Mon, 02 Apr 2012 00:00:00 +0200

Log4r wrapper with extra features for roles and sane error handling

bio-gngm 0.1.0 released

Mon, 19 Mar 2012 00:00:00 +0100

Next-Generation Mapping of Mutations

bio-alignment 0.0.6 released

Sat, 17 Mar 2012 00:00:00 +0100

Support for multiple sequence alignments (MSA)