Scribblings on Embedded Systems

Shutting down power to USB hosts on BeagleBoard-xM

2011-05-28T02:02:00.000-07:00

So one of the things I want to do as a part of an idea one of colleagues suggested is to shut down power to the USB controller on my beagleboard-xM.

There is a single chip in the beagleboard that provides power to the entire board (T
MS65950) which provides power management apart from other things. The beauty of the design is all the supplies to this chip can be controlled over the I2C bus by the main processor.

Architecture:
Processor -> USB PHY -> USB HUB -> USB ports

There are 2 main power sources for 2 chips that are required for USB function.

1. A 1.8v supply coming from directly from the power manager (TMS65950) to power the USB PHY. Why is a PHY required? Because the processor on its own cannot speak USB. It is like an ADC if you will.

2. A 3.3v supply that powers the USB hub (SMSCLAN9514). This comes from an LDO which derives its power directly from the 5v rail (independent of the power manager). The beauty of the design is even though the LDO derives its power directly, it can still be controlled by the power manager which is inturn controllable over I2C bus by processor.

Because the power manager has different power supplies for different uses (like for processor and peripherals), the processor can selectively shutdown power to different components. This is quite awesome. I guess this makes sense for an embedded board which would like to turn off power supplies that are not required.

Infact- if you look at a PC architecture, it is not possible to shut down power to USB from kernel (if I'm right) unless your USB controller supports it. Not all devices support USB suspend mode and even if they did, the USB specification only says that they shouldn't draw more than a little current but technically no one is stopping them from drawing more :) In the case of beagleboard, we are strictly shutting down power supply.

Next thing I have to figure out is how to do all this in software :)

Thanks for reading, more updates coming.

BeagleBoard-xM

2011-05-19T22:56:00.000-07:00

Just got a BeagleBoard-xM to play with and some time now that my exams are over (finally).

It is a pretty nice board I must say, and I had it up and running in 15-20 minutes once I found the right cables in my lab.

I'm a fan of X-windows so I decided to create a new X display just for the beagleboard and made Gnome running on the board connect to it over the wired network. It was fun!
Here's a little demo:

The performance of the board is amazing, considering it is tiny and fanless. Thanks to the low power of the OMAP3 processor, it can be powered completely through a mini USB cable. The processor is at 1GHz and it comes with plenty of RAM allowing you to run a complete desktop environment and a few applications.

There is also a fantastic opensource community and the documentation is fairly well written though its hard for anyone to keep it uptodate as the board evolves.

Storage
The microSD card that came with it has all the images you need to test the board, though I am compiling my own kernel and U-boot to get my hands dirty. :)

It doesn't come with any NAND flash though so it relies on bootloading from a microSD card. If I'm right - the Boot ROM expects the boot partition to be FAT32 (I guess because reading FAT32 takes little code) from where it loads X-loader to kick start the boot loading process.

Ethernet and USB
The -xM comes with an integrated USB hub (SMSC LAN514) chip with support for 4 USB ports. The chip also has integrated Ethernet support with a complete 802.3 MAC and PHY. Isn't that cool!

Obviously there is a lot more to explore. More information: http://www.beagleboard.org/

Extra Notes:
1. How to get an X-windows session with the beagleboard:

On the X-server:

/usr/bin/X :1 -br -v      # X server instance on display 1
DISPLAY=:1 xhost +        # Permission for the board to connect

On the Beagle Board:

DISPLAY=joel:1 gnome-session # DISPLAY = X-server's host:display-name

Next:
I'm going to try building my own Xloader, Kernel and U-boot. I will most likely boot from NFS as I don't like ejecting and copying stuff to the microSD card which is quite tiny for my fingers. :)

An introductory guide to linking and loading

2010-06-12T23:36:00.000-07:00

So I decided to remove the post on dynamic linking because it was largely useless to everyone but myself, and I wanted to get into some serious writing on linking and loading mechanisms, and needed an excuse to learn LaTeX. ;-)

So I'll be working on a guide I'm calling "Gentle Introduction to Linking and Loading"

Will take material from the previous posts, edit it and add some more with nice examples and figures. This is a work in progress and will probably take a very long time to complete.

You can pull LaTeX sources of the work here or read it online. Ofcourse its all bits and pieces for now but I want to get some early feedback.

Your comments are much appreciated :) and if there's anything missing or you would like to contribute any part of it, I would love to hear from you.

Thanks

C Programming: Understanding function calls

2009-11-22T01:46:00.000-08:00

Today I had some fun playing with assembly to understand the mechanics of function calls and parameter passing on x86.

The C Program:

#include <stdio.h>
void func1(int a)
{
        printf("hey there, I'm in func1! %d\n", a);
}

void func0()
{
        printf("hey there, I'm in func0!");
}

int main()
{
        func0();  /* Pass no arguments */
        func1(1); /* Pass 1 argument */
        return 0;
}

Next I compiled and disassembled this:

gcc -o func.c func
objdump -d -M intel func > func.dump

Disassembled Output (heavily commented):

;;;;;;;;;;;;; func1 disassembled ;;;;;;;;;;;;;;;
080483c4 (func1):
 ; Save the calling func's stack frame pointer.
 80483c4:       55                      push   ebp 

 ; Create the stack frame pointer for this function. This will be used later as
 ; a reference point from which the arguments that were just pushed to stack
 ; from the main function can be retrieved (see main function below)
 80483c5:       89 e5                   mov    ebp,esp  

 ; Allocate space on the stack for local variables.
 80483c7:       83 ec 08                sub    esp,0x8 

 ; Fetch the argument passed from calling function (integer 1)
 ; We add 8 to ebp because we need to go back in the stack to a point where
 ; we can fetch the argument being passed. Its 8 because we used 4 while
 ; pushing ebp (above) and 4 more because the calling function pushed the
 ; address of the instruction to be executed after this function returns.
 80483ca:       8b 45 08                mov    eax,DWORD PTR [ebp+0x8]

 ; Store the fetched argument on the stack (to be passed to printf).
 80483cd:       89 44 24 04             mov    DWORD PTR [esp+0x4],eax

 ; Store the address of the string on the stack (to be passed to printf).
 ; This string is stored in the ".rodata" section of the executable file.
 80483d1:       c7 04 24 f0 84 04 08    mov    DWORD PTR [esp],0x80484f0

 ; Call the function to format/print the text.
 80483d8:       e8 1b ff ff ff          call   80482f8  

 ; Restore the frame pointer (ebp) to its original value and return.
 80483dd:       c9                      leave    
 80483de:       c3                      ret    

; func0 is the same as func1 except we don't need to fetch the integer
; and store it on the stack.
080483df (func0):     
 80483df:       55                      push   ebp 
 ; .. skipped ..

;;;;;;;;;;;;; main function disassembled ;;;;;;;;;;;;;;;
080483f3 (main):
 80483f3:       8d 4c 24 04             lea    ecx,[esp+0x4]
 80483f7:       83 e4 f0                and    esp,0xfffffff0
 80483fa:       ff 71 fc                push   DWORD PTR [ecx-0x4]
 80483fd:       55                      push   ebp
 80483fe:       89 e5                   mov    ebp,esp
 8048400:       51                      push   ecx
 ; Allocate space on stack for local variables/args.
 8048401:       83 ec 04                sub    esp,0x4

 ; Call func0 (no arguments, nothing on our stack).
 8048404:       e8 d6 ff ff ff          call   80483df 

 ; Store the integer 1 on stack which will be retrieved by the function func1.
 8048409:       c7 04 24 01 00 00 00    mov    DWORD PTR [esp],0x1 
 8048410:       e8 af ff ff ff          call   80483c4 

 ; This I'm guess is the return value of the main funciton.
 8048415:       b8 00 00 00 00          mov    eax,0x0

 ; Relinquish allocated stack space.
 804841a:       83 c4 04                add    esp,0x4     

 ; Restore the contents of registers and the stack frame pointer (ebp).
 804841d:       59                      pop    ecx     
 804841e:       5d                      pop    ebp
 804841f:       8d 61 fc                lea    esp,[ecx-0x4]

Learning assembly is fun and can further your understanding at a much deeper level.
I highly recommend Programming from the ground up for an introduction to programming, assembly and linux.

Functions with Pattern Matching arguments in Common Lisp.

2009-02-28T21:27:00.000-08:00

I recently stumbled upon a cool CL library that makes it possible to define functions with pattern-matching arguments (like how its done in Haskell/Erlang). This is particularly useful and elegant, there are numerous benefits to your Language supporting pattern matching function arguments. The most important according to me are conciseness, elegance and clarity. Slava articulates this very well here.

Why I love pattern matching:
- Pattern matching makes it very easy to write recursive programs. For example, for terminating the recursion, traditionally you'd have to wrap your terminating condition (for example, (equal list1 nil)) in a cond or if, and manually test it during every recursive call. That's something you'd hate doing if you've ever written programs in Erlang or Haskell.
- I frequently write programs that progressively destructure lists, and having to pull out the car of a car of a car, can be a pain sometimes. With pattern matching, I can have these pulled out and bound to variables - automatically.

Unfortunately, Common Lisp doesn't have such a notion built into it (some people talk about Destructuring bind. Please, it works only for macro parameters, not functions. and is very limited in what it can do (in this context)).

To see just how nice pattern matching is and why it would suck if your language didn't or couldn't have it, consider the classical factorial example in Common Lisp and its erlang equivalent:
Common Lisp:


(defun fact(n)
  (cond ((equal n 0) 1)
        ((> n 0) (* n
                    (fact (- n 1))))))

Erlang:


fact(0) -> 1;
fact(N) -> N * fact(N - 1).

Erlang clearly wins, the code is much more readable and elegant than the common lisp version. Lisp with an evil grin, says, hold it right there! and pulls out a cool library from its arsenal called bpm. Its really cool in that its written entirely in Portable Common Lisp and is yet another example of how powerful lisp can be in extending itself. With bpm, you can now define your lisp functions with arguments that pattern match and bind to variables.. just like how you would in Erlang/Haskell.

Some examples in Common Lisp to show you how clean your code will look:
Factorial:


(def! fact (0)  1)
(def  fact (_n) (* _n
                   (fact (- _n 1))))

Fibonacci of a number:


(def! fib (0) 0)
(def  fib (1) 1)
(def  fib (_n) (+ (fib (- _n 1))
                  (fib (- _n 2))))

It gets even better when you have to destructure lists recursively:
A function that adds all the elements of a list:


(def! myf1 () 0)
(def  myf1 (_head . _tail)
  (+ _head (myf1 _tail)))

Sweet! :)

Erlang: A Dynamic Controllable Timer

2009-01-25T06:19:00.000-08:00

Here's a modified version of the Timer example in Joe Amstrong's great book "Programming Erlang"
Joe talks about spawning processes and receiving messages with timeouts, and then goes about building a timer using these concepts. This version makes it a bit more dynamic and self-changing according to messages you send the timer once you create it.
It also shows how you can use recursion to build a periodic timer..

Code:


-module(xtimer).
-export([make/2]).

make(Period, Message) -> spawn(fun () -> start(Period, Message) end).

start(Period, Message) ->
    receive
        NewMessage when is_list(NewMessage)  -> start(Period, NewMessage);
        NewPeriod  when is_number(NewPeriod) -> start(NewPeriod, Message)
    after Period ->
        io:format("~p~n", [Message]),
        start(Period, Message)
    end.

Explanation:
1. The first 2 lines are self-explanatory
2. The make function is what we call from the shell and it spawns a timer which prints Message every Period milliseconds. Make also returns the timer's Pid which we use to control it.
3. We then send messages to our timer using its Pid to change its behaviour (Message/Period).

everytime you need to change the timer properties, No need to stop and restart, just send it a message! try doing that in any other language :-)

Test run..


Eshell V5.5.5  (abort with ^G)
2> Timer = xtimer:make(2000, "Joel loves erlang!").
<0.38.0>
"Joel loves erlang!"
"Joel loves erlang!"
"Joel loves erlang!"
"Joel loves erlang!"
"Joel loves erlang!"
3> Timer ! "Timer message changes....".
"Timer message changes...."
"Timer message changes...."
"Timer message changes...."
"Timer message changes...."
"Timer message changes...."
4> Timer ! 3000.     % Timer is now slower...
3000
"Timer message changes...."
"Timer message changes...."
"Timer message changes...."
5> q().

stumpwm mail/message notifications from Emacs Jabber - walkthrough

2008-11-09T08:50:00.000-08:00

Over the weekend, I got to install the stumpwm window manager on SBCL, and its absolutely brilliant. A tiling, fully keyboard controlled extensible window manager.
I also got to get SLIME up and running which gives me direct access to the running stumpw process, I can directly eval code from emacs/slime and change the behavior of the window manager, very neat indeed.

For starters, I added all a lot of stuff to the mode line, with my hostname, battery charge, date, IP address etc. The best part, I was able to make emacs talk to stumpwm through slime, so I thought I might as well go ahead and integrate jabber.el and jabber-mail-notify.el (which notifies emacs of new mail) to the stumpwm modeline and message bar.

The initial model I adopted was

Emacs Jabber -> Slime -> Stumpwm (swank loaded) -> Stumpwm modeline/message-bar.

The drawback was I always had to startup swank and slime-connect to it from emacs before sending any messages to stumpwm. So I adopted a flat file approach (emacs talking to stumpwm through flat files), which turned out to be very quicky and efficient.

Emacs Jabber -> File write -> Stumpwm File read -> Update sumpwm modeline/message-bar.

The result was neat and working (see screen shots at the end)
Here's how I did it:
Firstly you will need jabber-mail-notify.el (an extension I wrote last week to notify emacs jabber of new google mail messages).
Once you have emacs jabber, with jabber-mail-notify.el in place, add the following to your emacs de-jabber.el init file:
(this code registers 2 emacs jabber hooks, one for new chat requests, and the second for new mail.)


;; Stumpwm new message mode-line integration
(defun jabber-stumpwm-activity-modeline-update ()
  (let ((text jabber-activity-mode-string))
    (if (not (eq (length text) 0))
        (setf text (concat " [Message from " text "]")))
    (write-string-to-file "~/emacs-jabber.temp" text)
    (princ "Done.")))
(add-hook 'jabber-activity-update-hook 'jabber-stumpwm-activity-modeline-update)

;; Stumpwm new mail mode-line integration
(defun jabber-mail-stumpwm-mode-line-update(iq-data)
  (let* ((text (jabber-build-mailbox-string iq-data)))
    (write-string-to-file "~/emacs-jabber-mail.temp" text)
    text)
  (princ "Done."))
(add-hook 'jabber-mail-notification-hook 'jabber-mail-stumpwm-mode-line-update)
;; -------------------------------------------

Now lets make stumpwm periodically read the temporary files for any mail/chat updates using the unix cat command.
Add the following to your ~/.stumprc or wherever you initialize your mode line format:


(in-package stumpwm)
(defun show-emacs-jabber-new-message ()
  (let ((new-message (run-shell-command "cat ~/emacs-jabber.temp" t)))
    (substitute #\Space #\Newline new-message)))

(defun show-emacs-jabber-new-mail ()
  (let ((new-mail (run-shell-command "cat ~/emacs-jabber-mail.temp" t)))
    (if (not (eq (length new-mail) 0))
        (progn (stumpwm:message new-mail)
               (run-shell-command "rm ~/emacs-jabber-mail.temp" t)
               (run-shell-command "touch ~/emacs-jabber-mail.temp" t)))
    ""))</PRE>Lastly modify your *screen-mode-line-format* variable such that it includes the following bolded elements at the end of the list.<PRE>(in-package stumpwm)
;; Set mode-line format
(setf *screen-mode-line-format*
      (list
       '(:eval (show-hostname))
       "| Battery:"
       '(:eval (show-battery-charge))
       '(:eval (show-battery-state))
       "| IP " '(:eval (show-ip-address))
       "| " '(:eval (run-shell-command "ruby -e \"print Time.now\"" t))
    '(:eval (show-emacs-jabber-new-message))
    '(:eval (show-emacs-jabber-new-mail))))

That should make your modeline periodically call those methods you just added. ..Done. restart stumpwm and emacs jabber, and you should be good. If you're smart, you can use slime, and ielm, and eval everything on the fly, without any restarts. That's the beauty of emacs and stumpwm! dynamic runtime evaluation is beautiful.
With the above code in place now, I can receive notifications of new mail, and chat messages from any window/frame/group of stumpwm. This is very handy for me as I no longer would silently ignore chat/mail notifications from emacs jabber just because I was on some other window/group.
If you hit any problem, feel free to mail or message me, now that I have the stump / emacs-jabber integrated, I will instantly get your message. :)

Screen shots (click to view larger):

stumpwm modeline getting updated with a new chat notification by emacs-jabber:

stumpwm message bar updated with a new mail message notification by emacs-jabber:

stumpwm message bar updated with a list of unread email on connecting to jabber:

my first emacs extension - A google mail notifier for Jabber

2008-11-01T23:44:00.000-07:00

Google talk's XMPP protocol has a few nice extensions to the standard XMPP, one of them being new mail notifications. So for all those google talk lovers, now you can receive email notification, using my new emacs extension (very-very-easy-to-install):

http://atlantisbangalore.selfip.com/~joel/projects/jabber-mail-notify.html

Hacking an Art

Hacking is an art, where you take existing stuff, reuse it, add your own stuff to it, read docs, manuals whatever you can get your hands on, look for pieces of information and code examples that are critical to solving your problem or accomplishing your objective. Most important of all, having fun while doing it!

For the emacs jabber extension, the following helped me a great deal:
- The Emacs jabber docs
- Google talk's XMPP extensions
- The most powerful development environment (Emacs !!)
- A knowledge of Common Lisp

The power of emacs as an emacs-lisp development environment.

The fact that emacs allows you to eval code that changes its behavior on the fly while its running is simply a big win. Its simply something that has led to its success, and the huge emacs code base that there is today.

Bottom Up programming approach.

For every function that I wrote, I was able to test whether it did what it was supposed to by evaluating it at point. (C-x C-e). That way, I was confident as I wrote the program, that I wouldn't need to go back and debug it. I even forgot about the code as I wrote it, and as it got progressively more complicated. The reason I found it okay to forget about the code I had already written was I wrote the extension using a bottom up programming style.

The 2 pieces of the puzzle

I independently worked on 2 separate parts of the extension.
1. Getting google to send me notifications and understanding the response.
Fetching new unread mail.
2. Parsing google's XML response and formatting it for display.

The first part: The first part was easy thanks to the dynamic emacs programming environment, which allows you to eval code on the fly. By writing dummy samples of code, I was able to understand how the whole XMPP communication works, the data that google sends to me and that it expects me to respond with. I was able to test my functions, on the fly and see google's jabber server response in the emacs jabber xml logs. It felt like a pilot, pushing the joysticks and seeing the plane respond. That is what I call fun!!

The second part: Fetching the data was a small part of the deal. The XML returned by google is fairly complex, and I had before me the myriad task of parsing that huge bunch of gibberish thrown at my screen! I will spare the details, but trust me it really looked ugly. :-) Lisp being a functional language - bottom-up programming is natural and easy, and perfectly suited for the parsing problem at hand!
For parsing the email messages, I wrote functions to extract the subject, the sender-list, the body, etc, and these were inturn built using smaller functions. Bringing all these methods together, I was able to write complex working code in just a few hours. Testing and evaluating each function as I wrote it, part by part, building bigger ones from the smaller ones, and making sure that it worked at every step (oh its really peaceful when you're confident that what you've written so far works)

Once these 2 major parts of the extension were written and tested, all I had to do was connect the dots. And yes, as you expect, it worked the first time. These little programming experience make me a better programmer, and that is the whole motivation behind these adventures, besides the fun ofcourse.

A 3 level Nested SQL Query on a single table.

2008-10-26T05:45:00.000-07:00

I had a lot of fun today writing the most complicated SQL query I've ever written.

The table given to me was something like this:

  CUSTOMER          PRODUCT           SALE_DATE     STATE     QTY
  -----------------------------------------------------------------
  Bloom              Pepsi           1999-02-04      NY       2233
  Bloom              Egg             1996-05-06      NJ       1233
  Abhay              Pepsi           1987-06-07      NJ       2222
  Bloom              Pepsi           1983-03-04      NJ       122

Question:
For each combination of customer and product, output the maximum
and minimum sales quantities along with the corresponding dates
(i.e., dates of those maximum and minimum sales quantities).
--
My first solution:
Ah, I said, .. and wrote the following expression after scratching my head for a minute

SELECT MIN(QTY), MAX(QTY) FROM CUSTOMER_TABLE_NAME GROUP BY CUSTOMER, PRODUCT;

But then.. reading the question carefully, I noticed that I had to give my result 2 dates, dates of those maximum and minimum sales quantities.

So I scratched and scratched my head, and finally when I couldn't scratch any more, I wrote the following query, which is the most complicated I've ever written.. but it works very well and has rewarded me with a complete SQL refresher. The solution is below, I am using 3 levels of nesting with implicit inner joins, at various levels.
Final Solution:

SELECT min_table_2.customer, min_table_2.product, max_table_2.maxq2, min_table_2.minq2,
 max_table_2.sale_max, min_table_2.sale_min

 FROM (SELECT customer as c2, product as p2, sale_date as sale_min, min_table_1.minq as minq2
     FROM  (SELECT customer as c1, product as p1, MIN(qty) as minq
             FROM sales GROUP BY customer, product)min_table_1,
             sales
         WHERE  min_table_1.c1 = customer AND min_table_1.p1 = product AND min_table_1.minq = qty)min_table_2,

     (SELECT customer as c2, product as p2, sale_date as sale_max, max_table_1.maxq as maxq2
     FROM  (SELECT customer as c1, product as p1, MAX(qty) as maxq
             FROM sales GROUP BY customer, product)max_table_1,
             sales
         WHERE  max_table_1.c1 = customer AND max_table_1.p1 = product AND max_table_1.maxq = qty)max_table_2

 WHERE min_table_2.c2 = max_table_2.c2 AND min_table_2.p2 = max_table_2.p2

Reasoning:
The outer most FROM joins the inner 2 tables (that we generate).
First we generate one result set from a subquery, the columns of this result set are: customer, product, MinQty.
But, we need the date of the record with the minimum quantity, so we use write another subquery using the results of the inner most query, and generate a new result set with the following columns: customer, product, MinQty, min_date
We similarly generate another table for maximum quantity with columns: customer, product, maxQty, max_date.
Once this much is done, we join these 2 generated result sets with the outer most query, the criteria for this implicit inner join is the customer and product columns of the minimum and maximum qty result sets should be equal. This is specified in our outer WHERE clause

WHERE min_table_2.c2 = max_table_2.c2 AND min_table_2.p2 = max_table_2.p2

Once the outer level join is done, its just a matter of selecting the columns we need for the final result set.

SQL queries can be real fun :)
cheers!

Ruby on Rails - Around filters

2008-08-24T01:15:00.000-07:00

In this article, I will show you how you can use RoR around filters, with exception handling to keep your controllers clean and do away with redundant chunks of code.

The problem we're going to discuss is writing repeated code in your controllers to perform validation on ActiveRecord objects (making sure they're non-nill most of the time), and then keep performing the same validation on all derivates of this non-nil object.
And it does not end there, you have to perform the same validation in all actions that use the same active record object and its derivatives.

To give you a better idea of what I'm talking about, here's a small snip that suffers from the very same problem:

Code from a BadPolicy controller:

class BadPolicyController < ActionController::Base
 def getpolicy
  if user = User.find_by_name(params[:name])
   if policy = user.policy
    render :text => "Policy for user: #{policy.attributes}"
   else
    render :text => "Policy does not exist" # VALIDATION 1
   end
  else
   render :text => "User doesn't exist"  # VALIDATION 2
  end
 end
 def deletepolicy
  if user = User.find_by_name(params[:name])
   if policy = user.policy
     policy.destroy
     render :text => "Policy for deleted succesfully"
   else
    render :text => "Policy does not exist" # REPEATED
   end
  else
   render :text => "User doesn't exist"  # REPEATED
  end
 end
end

Eek! we're performing the same validations on a user, and on his policy (derived from a user object) twice, that's 4 validations in 2 actions, and the situations gets worse when we add more actions to the controller. 8 validations in 4 actions, 16 in 8 and so on!
Patterns are a bad sign in programming, not only in aesthetic terms but in terms of hard-to-debug programs and increased development time.

Here's what you could do, or rather should do:
Before that, let me talk a bit about around filters which we'll be using to solve the above problem.
Around filters allow us to define methods that wrap around every action that rails calls. So if I had an around filter for the above controller, I could control the execution of every action, execute code before calling the action, and after calling it, and also completely skip calling the action under certain circumstances if I wanted to.

Here's an example of the around filter from the Rails API documentation.

 around_filter :catch_exceptions
 private
  def catch_exceptions
   yield
 rescue => exception
  logger.debug "Caught exception! #{exception}"
 raise
end

You can learn more about Around filters on the same page if you will, but my explanation here should suffice.

Ok now lets get to work. In order to kill redundant validation code in every controller action, we will:
(1) Wrap all the action invocations with our Around filter.
(2) Define new methods in the model code that wrap auto-generated rails methods such as "policies" and "find_by_name".
(3) Raise exceptions from within our newly created methods whenever required.
(4) Catch these exceptions in our Around filter, and process them.

First lets create a few custom exceptions that we'll be raising and catching:
Create a new file in the lib with any meaningful name, mine's called 'exception.rb' and add the following code to it:

# This our custom exception
class InvalidDataEx < RuntimeError
end

Now lets wrap a few auto-generated rails methods in the User model with our own.
Code for the User model, is as below. Note that exceptions are raised wherever required.

class User < ActionController::Base
 def self.named(name)
  find_by_name(name) or raise InvalidDataEx, "User doesn't exist"
 end
 def get_policy(name)
  self.policy or raise InvalidDataEx, "Policy doesn't exist"
 end
end

Code for a Good policy Controller:

class GoodPolicyController < ActionController::Base
 def getpolicy
  user = User.named(params[:name])
  render :text => "Policy for user: #{user.get_policy.attributes}"
 end
 def getpolicy
  user = User.named(params[:name])
  user.get_policy.destroy
  render :text =>   "User policy deleted succesfully"
 end
end

Or even better, we can completely do away with the imperative programming style and write our actions in a functional style:

class GoodPolicyController < ActionController::Base
 def getpolicy
  render :text => "Policy for user: #{User.named(params[:name]).get_policy.attributes}"
 end
 def deletepolicy
  User.named(params[:name]).get_policy.destroy
  render :text =>   "User policy deleted succesfully"
 end
end

Imperative programs are about HOW things are done - "get user, validate user, get his policy, validate policy, render"
Functional programs are about WHAT has to be done - "render a user's policy"

Lovely !! We've come a long way.

Now lets wrap both the actions in our around filter:
add the following to the controller class:

around_filter :catch_exceptions
 private
 def catch_exceptions
  yield
  rescue InvalidDataEx => err
  render :text => "#{err}"
  logger.debug "Exception #{err} occurred."
 end

Awesome! That's about it.

What you should be left with after reading:
Controller actions are expected to fetch data from models/lib, and render them, its not their job to perform data validation (most of the time).
By centralizing logic, making sure that its in the right place, and reusing it whenever possible, developing complex applications can become fun, intuitive and easy. Also this style produces programs that have fewer bugs, are easier to maintain and that require lesser development time.

When you see patterns in your code, then there's something wrong somewhere and a language as powerful and flexible as Ruby will leave you with options to identify and elliminate such patterns - as you saw in the example above. All it takes is a bit of thought.

Thanks for reading, comments are welcome!

Metaprogramming keeps it neat: DSL creation techniques in Ruby - Part 1

2008-07-06T21:30:00.000-07:00

Hi!

Due to the lack of resources on building DSLs in Ruby on the internet, I've written a small guide of how you can use DSL creation techniques in Ruby to cut down on lot of redundant code, make it easier to maintain code and improve readability. Hell you could eaven write your own language, here's your chance to discover how Rake, RoR and other amazing tools are written, and maybe even write one of your own!

This is not a how-to steb-by-step guide on building a full fledged DSL, rather I will show you how you can apply DSL-implementation techniques to cut down on lots of redundant code by examples that get progressively more complex as we address problems or hit a dead-end.

I've taken care to make sure that everything mentioned in this article makes sense even to novices, that's why I haven't slapped complicated code right way. Instead, I've taken a piecemeal approach.

Rules of the game: Detecting when something is not right: When you see a lot of redundant code sitting around, somethings wrong somewhere. Why-TF should your rewrite pieces of code, when you can make the language do that for you! No I'm not talking about doing some functional programming. Relax, you'll find out soon enough.

Redundancy sucks: If you've coded in languages like LISP, redundancy would be something just unacceptable to you. I actually haven't done much of Lisp, but a friend (lisp guy) managed to drill enough sense into me to stop typing that shit over and over again, and figure out alternatives!

Learning how to apply K-I-S-S (keep it simple stupid). While applying KISS, we tend to keep it so simple (KISS again) that the application turns out to be complex as it grows! Not in terms of a single method or module, but the application as a whole: complexity in terms of debugging, maintaining etc. I'd buy 500 lines of moderately complex code any day over 1500 lines of simple stuff!

DSLs to the rescue. So what is a Domain specific language: Its a language that provides you with all the infrastructure, and functionality required to rapidly build applications for a specific domain, without worrying about the details.

RoR - An amazing MVC framework that exploits ever corner of metaprogramming in Ruby and allows you to build simple yet powerful database driven web applications.

Rake- a tool very similar to the unix make. Rake (written completely in Ruby) executes "Rakefiles" allowing you to automate stuff like RoR migrations, file generation etc.

Components of a DSL implementation:

a base library providing you with all the functionality required for that domain (like ActiveRecord in RoR if you will)
a domain specific instruction set, like:

"has_one", "has_many", "belongs_to" etc in Ruby On Rails
"tasks", "desc", etc in Rake
Now that you know what a DSL is. We're going to learn how its done in Ruby.

Assuming your have a fair amount of Ruby knowledge, I'll walk you through some advanced concepts that we're going to use to spice up and make our app more intelligent so that it does all the hard work for us.

A few Ruby concepts

Class vs Instance methods: Yea, I know you know this. But what I want to state here is classes themselves can contain code, independent of any method. In RoR controllers, when you say "has_one", you're actually executing a class method "has_one", that takes the arguments and generates a whole bunch of instance methods for you.

class Products
has_many :vendors
end

The class method :has_many generates the instance method :vendors that returns an Active record collection of all the vendors for that particular product.

The define_method class method: define_method comes handy when you need to dynamically create an instance method in a class, all you have to it pass it the method name and block, and your instance method is up and ready.

Example:

class Foo
  define_method :bar do
    puts "I'm from bar"
  end
end

is equivalent to:

class Foo
  def bar
    puts "I'm from bar"
  end
end

Ok you're all set. With a basic knowledge of Ruby and the above concepts, you're ready to know what it takes to build a DSL in Ruby.

A Basic Implementation:
Give me control over my method definitions!
Inorder to cut down on code, the first thing I want to do is to get control over my methods, Every time a method is called, I want some code executed before and after it. Wow, is that even possible you say? Lets apply the concepts we learnt in the last section to make this possible:

Instead of using "def" to defined methods, I'm going to create a keyword "defm" and give it some meaning. For this I'm going to create a class method defm and add some logic to it. Then I'm going to use defm to create my own methods, and control them!
Here we go:

class MyClass
 def self.defm(method_name, &code)
   define_method method_name do
     puts "Starting method #{method_name}"
     return_value = code.call
     puts "method returned #{return_value}"
  end 
 end# Now lets use defm we just created:
   defm :foo  do
     puts "I'm inside foo!"
     true
   end
end

x = MyClass.new
x.foo

Output:
Starting method foo
I'm inside foo!
method returned: true

First I create a class method defm, then I define a method :foo using defm by passing in the method name as a symbol, and a block (body of the method). Executing foo on an instance gives us the desired result.

Note that if the method definition of defm sitting in the Class itself is bothering, don't worry, we can always move it to a module. I'll show you how in the next section. Infact there are a whole bunch of methods coming up and we have to move it to the module, the class just defines methods using our newly created keywords, the implementation has to abstracted.

Wow! I can now call methods created using our new defm keyword (in DSL terms), execute code before they start, and process their return value (boolean true in this case). You see where we're going? all the logic specific to defm-created methods can now be written in self.defm, and is not repeated with every method definition! Remarkable.

Err.. not really.. there's a problem:

What if I have a bar method, and try to call foo from it:

defm :bar  do
 puts "I'm inside bar!"
 foo
 true
end

Trying to call x.bar throws the following error:
NameError: undefined local variable or method `foo' for MyClass:Class

Ruby complains because the block we're passing to defm is executing in the context of the Class in which we created it (so self = MyClass).

Now what?

That's where instance_eval comes in.
instance_eval is required to change the context in which a block is executed. In the above example I'm passing a block to define_method. The block was created with self as the class object Bar. Inorder to execute the block with "self" as the instance of Bar on which it is called, all I need to do is use self.instance_eval(&block) instead of block.call .. the block now executes with self as the instance of Bar, not the object in which we created it.

change:
return_value = code.call
to:
return_value = self.instance_eval(&code)

x.bar now gives:

I'm inside bar!

I'm inside foo!

Wow!
Well done.. You now have built a basic working app by applying some metaprogramming but there are a few problems here:

Problem 1:
If you look in terms of what is stored in memory, every method we define has the logic for
"starting method" and "method returned" repeated. Though its not repeated in the code, it is repeated in every method definition stored in memory.

This is what our "bar" and "foo" actually look like in memory as you might guess:

def bar
# code before method starts
 return_value = self.instance_eval(&code)
# code after method returns
end
def foo
# code before method starts
 return_value = self.instance_eval(&code)
# code after method returns
end

Ugh! That's redundant again! (though thankfully only in memory this time) -we'll fix this soon enough.

Problem 2:
What we're passing to defm is a ruby block. Blocks or Procs aren't really methods, they're a lot different. Trying to return a value in the middle using the return keyword will break it.

Try this:

> func = lambda { |x|
* return 1
> }
> self.instance_eval(&func)
LocalJumpError: unexpected return
from (irb):97
from (irb):99:in `instance_eval'
from (irb):99

Problem 3:
We also haven't talked about passing arguments to our newly created methods. Even if we did, methods created using "define_method" start to execute even when passed the wrong no. of arguments. Our current implementation isn't smart enough to catch this.

Try this, it will still work:

> class Object
> define_method :x do
* |x|
* puts x
> end
> end
=> #
> x
(irb):57: warning: multiple values for a block parameter (0 for 1)

Ruby only warns you, it doesn't raise an ArgumentError as it normally would with a real method. Blocks unlike real methods in ruby can be passed the wrong no.of arguments, this means our methods aren't as robust as real ones.

In the next part (which is coming soon), we're going to address all these issues. We'll also build a smarter defm implementation in a Rails controller to demonstrate how we used this technique to cut down on tonnes of redundant code.

Hope you enjoyed this so far, stay tuned for some more fun!