Author at Infinum

Things I Wish ActiveRecord Had After Using Ecto

Vladimir Rosancic — Tue, 16 Jan 2018 10:10:00 +0000

A few years ago, I wrote an article about moving from Django to Rails. As I said there, I don’t find it too hard moving from one programming language to another, as long as you have good knowledge on concepts they all share. Each language, of course, brings something on its own.

Last year, I decided to try something new. As a Ruby developer, Elixir proved to be a logical choice, and I have to admit, I am not disappointed. I fell in love with it at first sight, and I enjoy working with Elixir/Phoenix now.

One thing that I especially like is Ecto, a database wrapper, and a domain-specific language for writing queries in Elixir. Although it’s not entirely the same thing as Rails’ Active Record (Ecto is not ORM), they serve the same purpose.

In this article, I’ll write about few things I like about Ecto that aren’t available in Active Record. It’s not meant to be a “Phoenix is better than Rails” rant as both frameworks provide really great things and we can all learn from each of them.

Changesets

When you want to insert or update the data in your database with Ecto, you should use changesets. As the official documentation says, they allow you to filter, cast and validate changes before you apply them to the data. When I first saw the concept of changesets, I thought of a form object pattern that I often use in Rails. Changesets, like form objects, allow you to have different ‘forms’ for various use cases. Think of a registration form vs. update profile form. Both operate on the same model (User), but they have different validations.

def registration_changeset(struct, params) do
  struct
  |> cast(params, [:email, :password])
  |> validate_required([:email, :password])
  |> unique_constraint(:email)
  |> put_password_hash()
end

def update_profile_changeset(struct, params) do
  struct
  |> cast(params, [:first_name, :last_name, :age])
  |> validate_required([:first_name, :last_name])
end

To accomplish this in Active Record, you have to use conditional validations (if/unless) which is ugly and hard to reuse, or you would have to use external gems like Active Type or Reform.

Ecto provides it out-of-the-box. The whole data manipulation philosophy in Ecto relies on the concept of small, reusable and modular changesets making it a powerful tool.

Validations use database constraints

Ecto can transform database constraints, like unique indexes or foreign key checks, into errors. This allows for a consistent database while giving users proper feedback and friendly error messages.

Let’s say we have a unique index on email field in users table.

create unique_index(:users, [:email])

If we try to insert a user with an email that already exists, an exception will be raised with a message like this:


** (Ecto.ConstraintError) constraint error when attempting to 
insert struct:

    * unique: users_email_index

If you would like to convert this constraint into an error, please
call unique_constraint/3 in your changeset and define the proper
constraint name.

Let’s call the unique_constraint/3 function in our changeset.

def changeset(struct, params) do
  struct
  |> cast(params, [:email])
  |> unique_constraint(:email)
end

Now, if we try to do the same thing as above, Ecto will catch the exception and transform it into a readable changeset error:

changeset.errors
[email: {"has already been taken", []}]

On the other hand, Active Record does not catch errors raised by the database, but rather checks the database state right before performing a validation. This could lead to unwanted situations in case of high traffic and a lot of parallel requests.

Preloading

We all know that N+1 queries are a bad thing. In most cases, you want to lower the number of database queries in a single request. Both Active Record and Ecto provide a way to specify a list of associations that need to be fetched along with the main query.

The difference is that Ecto will enforce preloading if you want to use the associated resource (it will raise errors in case you don’t), while Active Record will load that resource at the time you use it. As I tend to forget to preload associations and because I don’t want bad things in my apps, I like Ecto’s enforcement.

user = MyApp.Repo.get(MyApp.User, 1)
user.company
#Ecto.Association.NotLoaded

Batch inserts

To reduce the number of SQL queries, you might need to import data in batches. A use case could be to process data from large .csv or .xls tables. For that purpose, we use SQL’s INSERT statement with multiple rows.

In Rails, we could use a gem like Active Record Import or similar, but there is no built-in solution. Although Active Record’s create method can accept an array of objects, it does not perform a batch insert, but rather inserts each record one by one.

Ecto provides a built-in function called insert_all/3 that performs batch inserts and it works well.

MyApp.Repo.insert_all(
  User, 
  [
    [
      first_name: "John", 
      last_name: "Doe", 
      email: "john.doe@example.com"
    ],
    [
      first_name: "Jane", 
      last_name: "Doe", 
      email: "jane.doe@example.com"
    ]
  ]
)

Repo.one

Sometimes you expect only one record to be returned by your query. If you get more than one, you know something is wrong. Ecto provides a function called one that will fetch a single result from the query. It raises an exception if more than one entry is returned.

Imagine we have a unique index on nickname field in users table.

Somewhere in our code, we have a query to find a user with the specified nickname:

query = from u in User, where: u.nickname == "mickey"
Repo.one(query)

As we know we have a unique index, we are expecting only one result.

After some time, our business logic changes, and we add another field to our unique index – group_id. Now we can have multiple users with the same nickname, but in different groups.

Let’s assume that we changed our index, but forgot to change the query in the code; when we query on a nickname that is the same for multiple users, Ecto will raise the following error:

** (Ecto.MultipleResultsError) expected at most one result but 
got 2 in query

If Repo.one didn’t raise the error on multiple results, we probably wouldn’t easily notice that something is wrong. Our business logic would continue to work with the first user returned by the database, not necessarily the right one.

If we compare this with Active Record, methods like find_by would return the first record matching the specified conditions.

Ecto query language

Last, but not least, I like the way queries are written in Ecto. They just feel natural and SQL-ish to me. Let me give you an example of a query written in Ecto:

from user in User,
left_join: company in assoc(user, :company),
where: user.age < 25 or user.age > 60,
select: %{
  first_name: user.first_name, 
  last_name: user.last_name,
  age: user.age,
  company: company.name
}

Isn’t that beautiful?

While there are some things I didn’t cover in this article, it’s a brief overview of things I love about Ecto. Feel free to comment about things you love about each of these libraries.

The post Things I Wish ActiveRecord Had After Using Ecto appeared first on Infinum.

How to Efficiently Process Large Excel Files Using Ruby

Vladimir Rosancic — Wed, 15 Jun 2016 12:30:00 +0000

Last year, I worked on a project in which I needed to parse and handle large .xlsx files. Some of those files had more than 200K rows. I was looking for a gem that could efficiently do the job.

I wanted a simple solution that would just iterate over each row, parse the contents of the row and save data in database tables.

Popular gems in that time were Roo and Spreadsheet. I also found a simple gem called Simple xlsx reader that was easy to use and had everything I needed. Except one thing.

When I called SimpleXlsxReader.open(file_path), memory consumption would go to 2GB and it would stay that way until the garbage collector cleaned it.

What I needed was a library that would read Excel rows in streams. At that time, none of the mentioned gems had that ability. I found a gem called Creek that does only that – stream parsing large excel files. It is simple and works well. Memory consumption while parsing these files was now under 200 MB, which is acceptable. In the meantime, Roo was updated and now it has the same ability.

Also, to reduce the number of SQL queries, I implemented saving data in batches – 1000 rows in 1 query. So, my method reads a thousand rows, makes an array of ActiveRecord objects and then saves them at once. I used the gem called ActiveRecord Import for this purpose.

class ExcelDataParser
  def initialize(file_path)
    @file_path = file_path
    @records = []
    @counter = 1
  end

  BATCH_IMPORT_SIZE = 1000

  def call
    rows.each do |row|
      increment_counter
      records << build_new_record(row)
      import_records if reached_batch_import_size? || reached_end_of_file?
    end
  end

  private

  attr_reader :file_path, :records
  attr_accessor :counter

  def book
    @book ||= Creek::Book.new(file_path)
  end

  # in this example, we assume that the
  # content is in the first Excel sheet
  def rows
    @rows ||= book.sheets.first.rows
  end

  def increment_counter
    self.counter += 1
  end

  def row_count
    @row_count ||= rows.count
  end

  def build_new_record(row)
    # only build a new record without saving it
    RecordModel.new(...)
  end

  def import_records
    # save multiple records using activerecord-import gem
    RecordModel.import(records)

    # clear records array
    records.clear
  end

  def reached_batch_import_size?
    (counter % BATCH_IMPORT_SIZE).zero?
  end

  def reached_end_of_file?
    counter == row_count
  end
end

Finally, this task is being executed asynchronously in a background job since it’s too heavy to handle in the web process.

The post How to Efficiently Process Large Excel Files Using Ruby appeared first on Infinum.

I Moved from Django to Rails and Nothing Terrible Happened

Vladimir Rosancic — Tue, 10 Feb 2015 07:14:00 +0000

Rails vs. Django, an age-old debate in the web-development community. In this article, I’ll try to cover my own experience of moving from a Python-based web framework to a Ruby-based one.

The story begins with me graduating from university. My first job was in a media company that runs a number of large web portals, news websites and other online publications. I worked on developing and maintaining those websites as a Django developer. As you can imagine, it was a great and challenging job for a fresh programmer. But just as I settled in and became comfortable working in Django, an opportunity arose to work at Infinum as a Ruby on Rails developer.

Getting started is weird

It was a tough decision for me, but I decided to be brave and accept new challenges. One of the most motivating things for me was the fantastic work atmosphere and culture here at Infinum. I decided to learn Rails, and now, after 4 months, I have no regrets about making the switch.

In the beginning, some concepts were a bit strange to me. The Rails “convention over configuration” philosophy caused some confusion in my head because there was too much “magic” going on. When people talk about “magic” in Rails, they usually refer to things that the framework does “on its own”, but which you are required to know about. This is described in this StackOverflow article:

Well, consider a couple bits of Rails “magic”: when you write a controller class, its methods have access to certain variables and certain other classes. But these variables and classes were neither defined nor imported by anything in the file of Ruby code you’re looking at; Rails has done a lot of work behind the scenes to ensure they’ll just be there automatically.

And when you return something from a controller method, Rails makes sure the result is passed along to the appropriate template; you don’t have to write any code to tell it which template to use, where to find it, etc., etc.

In other words, it’s as if these things happen by “magic”; you don’t have to lift a finger, they just happen for you.

I had to learn to be more careful about naming conventions and the directory structure because Rails relies on those things to work its magic.

Shortly after that, I started noticing many similarities to the Django framework and very soon I learned how to do things in Rails that I knew how to do in Django. Almost everything that’s possible to do with one framework is doable in the other as well. The time required to start working on real-world projects was significantly shorter because of my Django experience.

Writing more tests

What confused me at first later became the thing that I love about Rails. Rails’ magic results in higher productivity and a cleaner code, for example in controllers. Controllers can look really nice, short and sweet. I like the Rails RESTful orientation – routes and controllers are designed in a way that it is best for you to follow the REST architecture.

# in routes.rb
resources :orders, only: [:index, :show, :new, :create]

# orders_controller.rb
class OrdersController < ApplicationController

  def index
    @orders = Order.where(active: true)
  end

  def show
    @order = Order.find(params[:id])
  end

  def new
    @order = Order.new
  end

  def create
    @order = Order.new(params[:order])
    if @order.save
      redirect_to @order
    else
      render ’new’
    end
  end

end

At Infinum, I started to write more tests than before. Rails places extra emphasis on testing and soon I realized how useful tests can be. Many great gems are made in order to make testing simple and clean.

The Rails asset pipeline is a great solution for managing static files (CSS, JS). I like the fact that I don’t need to worry about concatenating and minifying them. In Django, there is no built-in solution for this problem.

What I miss from Django

There are some things from Django I still miss. For example, I miss Django’s way of database migration. When you want to make a change to some table, all you have to do is directly change the model file and start the migration. When you change the model, the corresponding table in the database also changes. This way, you have your model definition and fields in one place.

In contrast, in Rails you don’t change the model file, but generate a migration file for every change you want to make. You don’t have a list of model fields in the model definition and you cannot change it directly like you can in Django.

Here is an example of how you do database migration in Django:

# Author model before migration
class Author(models.Model):
    name = models.CharField(max_length=80)
    slug = models.SlugField(max_length=80)
    is_active = models.BooleanField(default=True)

# We want to add new field called image
class Author(models.Model):
    name = models.CharField(max_length=80)
    slug = models.SlugField(max_length=80)
    # we add image field on third place
    image = models.ForeignKey(Image, null=True, blank=True)
    is_active = models.BooleanField(default=True)

# Now we only have to start the migration from the command line
$ python manage.py makemigrations
$ python manage.py migrate

Another thing I miss is Django’s great admin interface that you get out of the box when you create a Django project and database tables. It’s of great help when you want to view or change some data.

Last but not least, I like Django’s structure based on apps. Every app is responsible for a specific behavior and segment of a larger system. Apps have their own models, views and templates and they can be connected to other apps. This way, you get cleaner structure when you have a big application and a lot of files and directories.

Conclusion

All things considered, both frameworks are really good and both of them are making the process of building modern web applications simpler. If you are willing to put in some time and effort, it’s not hard to switch from one to another.

Here’s some advice – try to focus on the upsides of the target framework. I got used to the Rails way of thinking and now I’m really enjoying the platform’s signature behavior – the ability to write very concise and clean yet very powerful code.

As I said in the beginning, I moved from one good framework to another, and nothing terrible happened. Try it yourself – it’s easier than you think.

The post I Moved from Django to Rails and Nothing Terrible Happened appeared first on Infinum.