What I Have Learnt Writing A Sas Parser

So I’ve just released v0.01-alpha of my SAS transcompiler to Python (Stan).

Here is just a list of things I’ve learnt :

• SAS very similar to a PL\0 language
• by statements are inferior to the split-apply-combine strategy
• pyparsing makes life very easy (compared with dealing with lots of regex)
• iPython magics are ridiculously easy to write
• writing Python packages isn’t that hard, but there is a lot of extraneous options

Some of the (many) things which are missing:

• Just about every proc you can think of … you can define your own as a “function”. I know strictly speaking they are not the same thing, but for now it will do. (proc sql coming next release)
• As stated above no by statements, and hence none of the related statements as well (like retain).
• if-else-then-do not implemented correctly

But of course you would want to see it in action. So here it is!

from stan.transcompile import transcompile
import stan_magic
from pandas import DataFrame
import numpy as np
import pkgutil


import stan.proc_functions as proc_func

mod_name = ["from stan.proc_functions import %s" % name for _, name, _ in pkgutil.iter_modules(proc_func.__path__)] 
exec("\n".join(mod_name))


# create an example data frame 
df = DataFrame(np.random.randn(10, 5), columns = ['a','b','c','d','e'])
df

%%stan
data test;
set df (drop = a);
run;




u"test=df.drop(['a'],1)\n"




exec(_)
test

if statements combined with do end statements were difficult to implement. Here is my current implementation of if-then-else control flow, (I’ll have to revisit if and do end statements in the future…)

%%stan
data df_if;
    set df;
    x = if b < 0.3 then 0 else if b < 0.6 then 1 else 2;
run;




u"df_if=df\ndf_if['x']=df_if.apply(lambda x : 0 if x[u'b']<0.3 else 1 if x[u'b']<0.6 else 2  , axis=1)\n"




exec(_)
df_if

# procs can be added manually they can be thought of as python functions
# you can define your own, though I need to work on the parser
# to get it "smooth"

df1 = DataFrame({'a' : [1, 0, 1], 'b' : [0, 1, 1] }, dtype=bool)
df1

%%stan
proc describe data = df1 out = df2;
by a;
run;




u"df2=describe.describe(data=df1,by='a')\n"




exec(_)
df2

The proc actually isn’t difficult to write. So for the above code it is actually just this:

def describe(data, by):
    return data.groupby(by).describe()

This functionality allow you to handle most of the by and retain cases. For languages like Python and R, the normal way to handle data is through the split-apply- combine methodology.

Merges can be achieved in a similar way, by creating a proc:

%%stan
proc merge out = df2;
dt_left left;
dt_right right;
on = 'key';
run;




u"df2=merge.merge(dt_left=left,dt_right=right,on='key')\n"




left = DataFrame({'key': ['foo', 'foo'], 'lval': [1, 2]})
right = DataFrame({'key': ['foo', 'foo'], 'rval': [4, 5]})

exec(_)
df2

heres an example showing how you can define your own function and run it (not a function that came with the package)

def sum_mean_by(data, by):
    return data.groupby(by).agg([np.sum, np.mean]) 


%%stan
proc sum_mean_by data = df_if out = df_sum;
by x;
run;




u"df_sum=sum_mean_by(data=df_if,by='x')\n"




exec(_)
df_sum