SAS Community Planet

JMP Blog: Visual Data Quality with Named Colors in JMP, Part 2

2010-03-10T18:29:00+00:00

After my previous exploration of colors and names revealed inconsistencies in the Wikipedia color data, I looked around for a more authoritative source of color names. No luck finding an oracle, but I did find another interesting data set.

Where the Wikipedia table provided color values for a given set of names, this data set provides names for given color values. Dolores Labs created a Mechanical Turk task to have people give names to the color swatches they were assigned to. The results are provided in a CSV file, which JMP opens without any trouble at all.

As you might expect, there are lots of duplicate names. Out of about 10,000 color values, 1,000 were assigned the name "blue." Conversely, sometimes the same color value would be assigned different names, such as "purple" and "violet." And since the names were entered by the workers, instead of picked from a list, there is some variation in spelling. Column Recode is really handy for fixing up some of these problems. Here is a subset of the names after applying Trim Whitespace and Title Case in Col : Recode.

Color names before and after

I also used Find and Replace to change "Drk" to "Dark" and other abbreviations. That still left about 1,500 unique names, most of them occurring once. I wanted to see how much variation there was in colors of the same name, so I first filtered out the names with fewer than 15 color values. To do that, I used Summarize on the color name column, which told me how many colors had each name. Then I could sort the table by the counts and select the rows with fewer than 15 colors. Since the tables are linked, that also selects those in the original table. After excluding those, I plotted the remaining in Graph Builder as Hue by Name.

The graph is long, so I'll leave it to the end, but if you look closely (click the graph to see it full size), you might notice things like a purple called green and a green called blue. The names are ordered by average hue, which sometimes doesn't work too well for the reds, which often legitimately straddle the edge between 0° and 360°. So this is another example where a visualization helps detect data quality issues.

StudySAS Blog: $UPCASEw. format

2010-03-10T17:37:36+00:00

We all know the importance of UPCASE function in handling the character case strings. But do you know that a format can let you do the same as the UPCASE function (upcasing the variables).$UPCASEw. format works similar to the UPCASE Function. It also does one more thing which UPCASE function doesn’t. i.e: $UPCASEw. format let you apply length to the variable. Remember that w specifies the width of the output field. Example:*********************************************************; data new;*convert it to uppercase;name="studysas blog"; format name $upcase.;newname=put(name, $upcase32.);*Put function let you apply $upcase format;run;**********************************************; *The length of the new variable newname will be 32.

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]

JMP Blog: Changing JMP Graph and Selected Text Colors (for Windows OS)

2010-03-09T14:00:00+00:00

At Sam Gardner’s March 4 Webcast on JMP Journals, we closed with a few basic unanswered questions about changing color settings in JMP for Windows. I promised the answers.

1. You can change most graph settings globally from JMP Preferences. For example, to change most graph attributes, including background color, marker size, marker style and color schemes, choose PREFERENCES>REPORTS, then make and apply your changes.

2. To change the background color of tables, choose PREFERENCES> WINDOWS SPECIFIC>WINDOWS BACKGROUND COLOR.

3. To change the color used to highlight selected menu options or table rows that correspond to areas of a graph that you highlight, you need to change a Windows setting. NOTE: Because these settings will apply to all Windows programs, you may want to note the defaults before changing.

From your Windows Settings Control Panel, select DISPLAY>APPEARANCE>ADVANCED. From the ITEMS box, chooses SELECTED ITEMS, and then choose your color. You may also need to change the FONT color. For example, yellow highlight was unreadable with my default white font, so I changed FONT color to black.

Looking for other tips? Try our searchable JMP Knowledge Base & FAQs.

NOTE: The blog of RTSL.eu: Project Plans in Excel - Tracking to Completion

2010-03-09T11:50:00+00:00

The series on maintaining a project plan and Gantt chart in Excel has been popular, and I've had a lot of queries about tracking progress. So, in this bonus post I’ll describe how to display tasks’ progress on the Gantt chart that was featured in the previous posts in this series. In addition, I’ll show how to highlight “today”. Alongside this paragraph (right) you can see what the result of this post looks like.

In the three previous posts in this series I described how to create a neat and simple Gantt chart, how to add dates to the day numbers, and how to group tasks. These three simple sets of steps have given the developer sufficient knowledge to quickly create a simple but effective Gantt chart that demonstrates the developer is in control of the project (without spending more time on planning than on delivery). Alongside this paragraph (left) you can see what the results of our previous efforts looks like.

Let’s start by inserting a column after E and heading it “%Done”. This is where you'll need to type values to indicate your progress. Then, after column N let’s add “Done” and “DoneEnd”. I’ve made the text colour of the latter two columns a semi-visible grey because they’re our working values and not of interest to the reader of the Gantt chart. If you see the picture below, you’ll see that I’ve also populated the %Done column with some values.

Now let’s populate the calculated columns. The working columns are not strictly necessary, but they’ll help illustrate the calculations that we’re doing. Firstly, let’s understand what we’re trying to achieve. Cell H3 represents progress on activity #1 on day 1. We’ll display a block in the cell if progress on the activity is equal to (or greater than) half a day’s effort. So, for activity #1 we can see that effort is complete up to the end of day 3; for activity #2 the effort is complete up to 2/3 of the way through day number 2. Since day number 2 for activity #2 is more than half complete we’ll put a block in that cell (but not day number 3).

Let’s set Done to show the amount of effort completed, i.e. %Done times Effort (so Q3 =D3*F3/100). We’ll use DoneEnd to indicate the cell location where the completed effort ends (so R3 =C3-1+(F3*D3/100)).

DoneEnd=1.6 in cell R4 tells us that the effort is complete up to 0.6 of the way through the cell that represents day 2. So we’d place in block in day 1 and we’d also place a block in day 2 because it’s more than half complete.

We’ve previously used conditional formatting to indicate the start and end dates, so we’ll use formulae to shows blocks for completed effort in appropriate cells. The formula for cell H3 is =IF(AND($C3<=H$2,H$2-0.5<=$R3),"n",""). Once we’ve spread this across the whole of the Gantt we get the following result.

Oh, and change the font for the Gantt area to Wingdings so the blocks appear as blocks rather than lower-case n characters! You can also change the text colour to something other than black.

So, we now have %Done bars (blocks) overlaying our basic start-to-end Gantt bars. But we’ve avoided doing this for the grouped bars. Why? Well, we can calculate %Done as the sum of Done for the group divided by total effort for the group, i.e. F5 =100*Q5/D5 (first, you need to set Q5 to =SUBTOTAL(9,Q3:Q4)), but if you add this as blocks in the Gantt you’ll see that the bars run from day 1 to day 4 despite the fact that the individual activities for group Alpha have not hit day 4 yet. This can be confusing for the reader of the Gantt, so I usually keep the calculation, but remove the progress bars, leaving the end result below.

In addition to copying D5 to Q5, I’ve deleted R5 (just for clarity, because we’re not using it). And I hope you realised you need to do similar things in the other grouped rows, so Q8 contains =SUBTOTAL(9,Q6:Q7), and Q9 contains =SUBTOTAL(9,Q3:Q7).

Finally, I said I’d highlight “today”. Well that's easy, select the date cells (H1 to O1), open-up Conditional Formatting and set the formula =IF(H1=TODAY(),1,0). I’ve chosen to use white text on a black background:

Here’s the final spreadsheet:

Neat!

the SAS-BI blog: Alert Notification from BI Dashboard 4.2? Include EMAIL Addresses in User Manager

2010-03-08T09:00:02+00:00

If you would like for the BI Dashboard to send you an alert via email, you must ensure that the email address is included for each user that will be receiving emails. This is located in SAS Management Console User Manager.

Instructions: http://support.sas.com/documentation/cdl/en/biwaag/63059/HTML/default/a003311779.htm
Example:

SAS Programming: An efficient macro for Stumps-Recursive Partition weak classifier

2010-03-05T17:55:57+00:00

In this post, I post an improved SAS macro of the single partition split algorithm in Chapter 2 of "Pharmaceutical Statistics Using SAS: A Practical Guide" by Alex Dmitrienko, Christy Chuang-Stein, Ralph B. D'Agostino.

The single partition split algorithm is a simplified version of Stumps-Recursive Partition, and is a weak classifier, usually used to form the base weak learner for boosting algorithm. This specific classifier seeks to separate the space into 2 subspaces for independent variables where each subspace has increasingly higher purity of the response classes, say 1 and 0. In the examples, Gini Index is used to measure purity/impurity.

The SAS example macro %SPLIT in the book (found @ Here) is for illustration purpose, and is so inefficient that it practically can't be used by industrial standard.

I modified this macro and made it usable in real business applications where millions of observations and hundreds of variables are more than common.

Improved Code:



%macro Dsplit(dsn, p);
/************************************************/
/* dsn: Name of input SAS data sets. All        */
/*        independent variables should be named */
/*        as X1, X2,....,Xp and be continous    */
/*        numeric variables                     */
/*   p: Number of independent variables         */
/************************************************/
options nonotes;
%do i=1 %to &p
proc sort data=&dsn.(keep=y w x&i)  out=work.sort; by x&i
run;

proc means data=work.sort noprint;
     var y;
     weight w;
     output out=_ysum  n(y)=ntotal  sum(y)=ysum;
run;
data _null_;
     set _ysum;
     call symput('ntotal', ntotal);
     call symput('ysum', ysum); 
run;
data y_pred;
     set work.sort  end=eof;
     array _p[&ntotal, 2] _temporary_;
     array _g[&ntotal, 2] _temporary_;
     array _x[&ntotal]    _temporary_;
     retain _y1  _oldx 0;
     if _n_=1 then _oldx=x&i
     _x[_n_]=x&i
     if ^eof then do;
        _y1+y; 
       _p[_n_, 1]=_y1/_n_; _p[_n_, 2]=(&ysum-_y1)/(&ntotal-_n_);
       ppn1=_n_/&ntotal  ppn2=1-ppn1;
       _g[_n_, 1]=2*(ppn1*(1-_p[_n_, 1])*_p[_n_, 1]+ppn2*(1-_p[_n_, 2])*_p[_n_, 2]);
       if _n_>1 then _g[_n_-1, 2]=(_oldx+x&i)/2;
       _oldx=x&i
     end;
     else do;
       _g[_n_-1, 2]=(_oldx+x&i)/2;
       ginimin=2;
       do i=1 to &ntotal-1
          gini=_g[i, 1]; x&i=_g[i, 2];
          keep gini x&i
          output;
     if gini lt ginimin then do;
        ginimin=gini; xmin=x&i
            p1_LH=_P[i, 1]; p0_LH=1-p1_LH;
              p1_RH=_P[i, 2]; p0_RH=1-p1_RH;
        c_L=(p1_LH>0.5); c_R=(p1_RH>0.5);
    end;
       end;
     end;  
     do i=1 to &ntotal
        if _x[i]=xmin then y_pred=c_L;
        if _x[i]>xmin  then y_pred=c_R;
        keep y_pred y w; output y_pred;
     end;
     call symput('ginimin', ginimin);
     call symput('xmin', xmin);
     end;
run;
data _giniout&i
     length varname $ 8;
     varname="x&i";
     cutoff=&xmin
     gini=&ginimin
run;
%end;
data outsplit;
     set %do i=1 %to &p
            _giniout&i
         %end;;
run;
proc datasets library=work nolist;
     delete _giniout:;
quit;
option notes;    
%mend;

/* pure stump rules */

%macro stump(dsn, p);
/************************************************/
/* dsn: Name of input SAS data sets. All        */
/*        independent variables should be named */
/*        as X1, X2,....,Xp and be continous    */
/*        numeric variables                     */
/*   p: Number of independent variables         */
/************************************************/
options nonotes;
%do i=1 %to &p
proc sort data=&dsn.(keep=y w x&i)  out=work.sort; by x&i run;

data _null_;
     set work.sort  end=eof  nobs=ntotal;
  retain _wsum _ysum;
  _wsum+w; _ysum+y*w;
  if eof then do;
     factor=ntotal/_wsum;
     call symput('ysum', compress(_ysum*factor));
  call symput('ntotal', compress(ntotal));
  call symput('factor', compress(factor));
  end;
run;

%put factor=&factor
%put ysum=&ysum

data _giniout&i
     array _p[2] _temporary_;
     array _g[2] _temporary_;
     array _x[1] _temporary_;
     retain ginimin   xmin  _y1  _oldx _cn ;
     if _n_=1 then do;
        ginimin=2; xmin=x&i _y1=0; _oldx=0; _cn=0;
     end;    
     do while (^eof);
        set work.sort  end=eof;
        _y1+y*w*&factor _cn=_cn+w*&factor
        if _cn=0 then _p[1]=0; else _p[1]=_y1/_cn; 
        if _cn=&ntotal then _p[2]=0; else _p[2]=(&ysum-_y1)/(&ntotal-_cn);
        ppn1=_cn/&ntotal  ppn2=1-ppn1;
        _g[1]=2*(ppn1*(1-_p[1])*_p[1]+ppn2*(1-_p[2])*_p[2]);
        if _n_=1 then _g[2]=x&i else _g[2]=(_oldx+x&i)/2;
        _oldx=x&i
        if _g[1]2 then do; 
        ginimin=2; xmin=(_oldx+x&i)/2; 
     end; 
     varname="x&i";
     keep varname  ginimin  xmin;
     output;
     stop;
run;
%end;
data outsplit;
     set %do i=1 %to &p
            _giniout&i
         %end;;
run;
proc datasets library=work nolist;
     delete _giniout:;
quit;
option notes;    
%mend;

Comparing the time used and results.

The example data used is the AUC Small training data, 200 sets of predictors for 15,000 ratings, from AusDM2009 competition, and can be found @ Here .

Using example code from the book, it takes 1563 seconds on a regular Windows desktop (Core2Duo E6750 2.67GHz, 4GB Memory, 7K2 rpm HDD with 8MB cache) to process 5 numerical continous variables, whereas with improved macro, it only takes 1 second to process the same amount of data. This improvement is criticle since weak classifiers like this one won't be used alone, but as the base for more time-consuming Boosting algorithms. With original macro, it is practically not usable for any boosting algorithms on data sets with hundreds of or more observations.

Original Macro:

Improved Macro:

Comparing the results from original macro and the improved macro, they both select X3 with the same partition cut off point and the same Gini Index.

Reference:
Pharmaceutical Statistics Using SAS: A Practical Guide by Alex Dmitrienko, Christy Chuang-Stein, Ralph B. D'Agostino, SAS Publishing 2007

The SAS Dummy: Just testing!

2010-03-05T15:57:43+00:00

Yesterday, we at SAS performed a little experiment, unintentionally. Who would notice (we wondered in retrospect) if we disabled one of our discussion forums on support.sas.com?

As we learned: a lot of people would notice.

There was a technical glitch that broke the SAS Enterprise Guide discussion forum for much of the day yesterday. All of the other discussion forums (and there are a bunch) were unaffected. But it warmed my heart to hear that so many people missed discussing SAS Enterprise Guide topics with their fellow users.

Thanks for participating in the forums. And thanks for letting us know that you missed us during the brief outage.

JMP Blog: Quick Reference Card

2010-03-04T20:28:36+00:00

Do you love keyboard shortcuts as much as I do? If so, be sure to check out the Quick Reference Card (found in JMP by going to the Help menu and selecting Books -> Quick Reference Card). It is a PDF of the keyboard shortcuts available in JMP, broken down by the operating system (Windows, Macintosh, or Linux). Many of them will be familiar to you because they are common keyboard shortcuts for the applicable operating system, but a number of them are unique to JMP. For example, did you know you can increase the font size by holding down the CTRL and SHIFT keys and then pressing the plus sign? Similarly, pressing the minus sign will decrease your font size. Also, did you know you can hold down the ALT key and press the first letter of a menu title to open that menu? The only exception is the Tools menu, which requires ALT+O. As another example, you can also move your current window to the back with CTRL+B. Do you need to create a new data table? CTRL+N. Do you need to show the log window? CTRL+SHIFT+L. Do you want to add the current report to a journal? CTRL+J. The list goes on and on…

In addition to keyboard shortcuts, the Quick Reference Card also features common clicks and special functions. Examples of some common clicks include holding the CTRL key down while resizing a graph so that you can resize all similar graphs the same way and right-clicking on a selected cell in a data table and selecting Fill to repeat values or a sequence of values in a column. Examples of special functions include how to check that fences match in a script (place the cursor to the left of an opening fence or to the right of a closing fence and both fences will highlight if a match is found) and how to select multiple analysis options from a red triangle menu (hold the ALT key down before clicking the red triangle).

Finally, you can find a detailed description of the options in the Formula Editor and the Tools menu. These components are especially helpful if you are a new JMP user who is learning to navigate the interface and learning what the different options mean. You might like to print a copy of the Quick Reference Card and have it handy as you learn more about JMP.

JMP Blog: See You at SAS Global Forum

2010-03-04T16:17:29+00:00

It’s like I won a golden ticket to Willy Wonka’s chocolate factory. That’s what it feels like to be a SAS employee who’s asked to go to SAS Global Forum, April 11-14 in Seattle. The buzz on the demo-room floor, hearing from thought leaders both entertaining and enlightening, the interaction between the brilliant developers and the genius folks who use the software … it’s really an action-packed few days.

But with that golden ticket comes a lot of responsibility. In addition to providing on-site coverage for this blog, I’ll be reconnecting with customers who have been using JMP for years as well as hosting members of the press who are just learning about JMP. But my self-imposed goal is to leave SAS Global Forum with no fewer than 10 new acquaintances, at least five of whom are using JMP and SAS together. I want to learn about their applications, and I want to turn those applications into success stories for the rest of the world to see.

So, if you’re using JMP and some other SAS product together, I want to meet you.

After you take in Stephen Few’s post-conference seminar and after you’ve been dazzled by my boss, Jon Weisz, at the Technology Connection, come find me. Seek me out between the papers presented by my JMP friends and colleagues, including my big boss, John Sall. When you build your agenda, build in time to share your story with me.

You’ll find me on the demo-room floor or wherever JMP papers are being presented. I look forward to seeing you in Seattle!

SAS Global Forum: SAS CEO Jim Goodnight Extends Special Invitation

2010-03-03T21:23:00+00:00

As Dr. Goodnight explains, SAS Global Forum (formerly known as SUGI) was started by SAS users, who held their first conference in 1976, several months before SAS was even incorporated as a company. The successful tradition of a user run conference, guided by an Executive Board of SAS users, continues today, 35 years later.

If you haven’t registered already, be sure to do so soon. Early registration ends March 8! Register now, and you can save up to $250 off your registration fee. There are additional ways to save as well. Fees will increase on March 9 for all Registration Packages.

This year will be my 14th conference, and I totally agree with how Dr. Goodnight sums things up in his invite video. “[SAS Global Forum is a...] huge collaborative forum where problems are solved, techniques are shared, and life-long friendships are made.”

SAS Global Forum: Michael Raithel's plans for SAS Global Forum

2010-03-03T20:57:00+00:00

While at SAS, I hope to get the opportunity to talk with all of the SAS icons. I’ve had the privilege to meet many. Before the SAS winter break, I had the privilege to talk with another: Michael Raithel.

At SAS® Global Forum 2010, Raithel will be presenting the Tuesday lunch feature presentation, “It’s not easy being a SAS programmer." So, I thought you might like to know a little more about who he is and what he does.

Who he is
Raithel is a SAS programmer, former NESUG conference chair, former SUGI section chair, three-time SAS author and a Senior Systems Analyst at Westat. Senior Systems Analyst … how many Senior Systems Analysts have you met? When I was at NESUG (my first regional), I met many. Raithel’s answer to my quip said it all, though. “If they called me ‘Senior Dog Catcher,’ it wouldn’t matter. I have a wonderful, fulfilling job here. This is an excellent place to work. There is relatively low turnover and very high job satisfaction; a very professional and serious, yet respectful, work environment.”

So far, he seems pretty much rank and file with the other SAS icons, doesn’t he? Raithel also has at least one other cool point: His first book, Tuning SAS Applications in the MVS Environment, is now part of the Smithsonian Institution’s Museum of American History Information Technology Collection. Well now, that’s the stuff of legends. Wouldn’t you agree?

SAS crunchers and munchers
Westat is an employee-owned corporation that provides contract research services to businesses, foundations, US government agencies, and state and local governments. A typical example of work Westat does is to host, manage and clean data for a state cancer registry. “SAS is an integral part of the software we use to manage the data” said Raithel. “Most of the other hundreds of research projects conducted for clients each year by Westat use SAS products.”

“We’re basically a meat and potatoes kind of SAS shop,” said Raithel. “By that, I mean we have 18 SAS products that are the foundation of much of our work. We have deployed four SAS/ACCESS® interfaces, SAS/CONNECT®, SAS/GRAPH®, SAS/Genetics™, SAS/IML®, SAS/Intrnet, and SAS/STAT®, to name a few. We are also looking to expand our use of SAS to include SAS® Data Quality.”

Westat may consider itself a “meat and potatoes SAS shop,” but the company is proving that treating people well leads to success. Success is promoted and encouraged by a support infrastructure that includes an in-house technical support unit and Westat-written SAS resources.

“On our intranet, we have SAS Resources Web pages that give our users one-stop shopping to learn what SAS is all about at Westat, including links to contact information for the people in my group, Westat’s SAS technical support,” said Raithel. “The pages include instructions for loading SAS on a desktop, SAS resources and Westat conference papers, documentation for the SAS products we use, the methods we use for validating hot fixes and even the date and location of the next Westat SAS Users Group meeting.”

According to Raithel, Westat’s SAS technical support department, which he heads, helps to ensure that SAS Institute's Technical Support doesn’t receive a flood of calls each day from Westat. Raithel and his two staff members “answer everything from, ‘I can’t open SAS today,’ to ‘What is happening with SAS 9.2?’ and ‘This report looks kind of funny.’”

When Raithel and his staff have questions they can’t answer, they usher them through the official SAS Technical Support process. “When there is a satisfactory result, and we believe that question is going to affect more users, we post it to the in-house listserv – SAS Outlook Information Forum,” he said. “SAS users at Westat also post questions and answers on the listserv.”

Success isn’t guaranteed, it’s mastered
Each year, Westat purchases SAS training EPTO units. “We buy enough EPTO units per year to allow us to have custom-tailored classes taught in our facilities,” said Raithel. “SAS Education has been very cooperative during the past three years in listening to our training needs and modifying some of the existing SAS classes to suit those needs.”

Westat also relies on a cadre of in-house SAS experts to teach beginner and intermediate SAS classes. “Once a year we have a SAS Institute speaker present at our Westat SAS user group meeting, and we invite speakers from outside organizations, such as the Federal Reserve Board and the Bureau of Labor Statistics,” said Raithel.

The education that Westat SAS users receive is not all piped in. Many Westat SAS users are experts in their own right. One former SUGI conference chair, two NESUG and one SESUG conference chair came from Westat’s staff. At Westat, SAS conference participation is indispensable. “We send a lot of people to SAS user group meetings” said Raithel. “During the past 13 years, Westat staff has published 200 SAS conference papers.

“We feel that conference experience is invaluable,” he said. “When our staff attends a SAS conference, they pick up tips from the presentations and a lot of good information comes back to Westat.

“After every conference, those who attend are on the hook for a 5- to10-minute presentation of the best papers and new ideas. We know that when we send people they’re going to enrich themselves by gaining new SAS knowledge, and that knowledge is going to come back and be available to other Westat staff.”

You can find some of Michael Raithel’s books, interviews, tips and past presentations on support.sas.com. You can meet Raithel at SAS Global Forum 2010 at the authors’ reception in the SAS Publishing demo area on Monday evening from 6 p.m. until 7:30 p.m.

Add “It’s not easy being a SAS programmer” to your SAS Global Forum agenda. This SAS Global Forum Tuesday lunchtime presentation by Raithel will take a lighthearted look at some of the societal, workplace and industry issues that SAS programmers routinely face. *An extra fee event.

The SAS Dummy: SAS: The . Manual

2010-03-03T13:32:37+00:00

Check out AnnMaria's blog covering what she's learned by being on the bleeding edge: using SAS 9.2 for Windows x64 on a Win7 virtual machine. Also, her short primer on XPT files makes me want to add some native capability into SAS Enterprise Guide.

If your goal is to send a SAS data set into a new JMP session, you can use this task right now. In SAS Enterprise Guide 4.3 (later this year), we'll have a built-in Send To JMP feature.

SAS from Out in Left Field: SaviDataSet 1.0.0.5 Released

2010-03-03T13:27:23+00:00

I just posted the latest version of the SaviDataSet reader/writer. The latest version allows for its use from the command line for converting SAS datasets to delimited, XML, and Excel files (2003 & 2007).

Please see the Help File in the Savian--> SaviDataSet --> Help.txt folder for instructions.

I will be incorporating a write feature in at some point but it is more complex since variables need to be defined.

The latest release can be found here:

SaviDataSet 1.0.0.5

the SAS-BI blog: Addressing Security at the Root - BI Dashboard 4.2 Tips

2010-03-03T09:01:00+00:00

You may choose to address security of who can view what dashboards within a SAS Information Delivery Portal page by managing the 'Sharing' capabilities at the Page or Portlet levels, however the most gradular level might also require a 'lock down' to avoid users surfing for and locating dashboards they shouldn't see.

In order to accomplish this, utilize the 'Folders' tab in SAS Management Console and navigate to:

System->Applications->SAS BI Dashboard -> SAS BI Dashboard 4.2

IMPORTANT NOTE: You will need to modify each element individually as the current version of BI Dashboard (9.2M2 - 4.2) doesn't support moving the elements into an organized folder structure.

Jurassic SAS in the BI/EBI World: HTML, PDF, RTF (and more) all at the same time.

2010-03-03T07:10:00+00:00

or . . . when is a session not really a session.

In my last post I illustrated explicity creating and using a session. Of course as with all things SAS related, that is not the only way to create a session!

In order for ODS to support creating html page with mixed text and graphics (e.g., a PROC PRINT and a PROC GCHART), ODS uses lightweight sessions. These are sessions without a lot of the overhead of a full session, but which allow a program to write some content, e.g., in the case of ODS, perhaps a graph where the generated HTML contains a link that replays the graph. So while it looks to the user like they are getting a table and graph in one request, there are actually two requests. Run this sample Table and Graph and take a look at the generated HTML for the img tag.

Note that this example is actually a SAS/IntrNet sample from my SAS Press Book . . . . . . but the Stored Process Server works the same way!

The generated link uses the replay program to do that. Upon a review of the generated HTML (just do a View Source in your browser) you see that the replayed graph is actually an entry in a SAS catalog. Well, it turns out that whenever you submit a request for the Stored Process Server to run a program, SAS creates a catalog and the catalog name is available to your program as the macro variable reference &_tmpcat.

And now it gets even more interesting because simply by writing something to that catalog causes SAS to create the lightweight session.

Now lets consider the case where we want to generate an HTML report, but you also want to provide a way for a user to access a printable version of the report (e.g., a PDF file) or an easily editable version (e.g., an RTF file) without having to rerun the reporting code. A lightweight session can be used to so that ODS will write PDF and RTF versions to the &_tmpcat catalog while generating/streaming the HTML version back to the users browser.

Just include statements like this in your program:

  filename pd catalog "&_tmpcat..shoes.pdf";
  filename rt catalog "&_tmpcat..shoes.rtf";

and then use the stpBegin macro for the HTML version and explicit ODS statements for the PDF and RTF versions:

  %stpBegin
  ods pdf file = pd style = sasweb notoc;
  ods rtf file = rt style=sasweb;

then include whatever your reporting code is (e.g., PRINT, TABULATE, REPORT, etc.).

We now need to include some HTML to provide the hyperlinks to that will replay the PDF and/or RTF versions. The following DATA steps creates macro variables whose values are the needed HTML:

  data _null_;
   call symput('replayPDF','''<center><a href="'||&_replay||'shoes.pdf">Printable (PDF)</a></center>''');
   call symput('replayRTF','''<center><a href="'||&_replay||'shoes.rtf">Editable (RTF)</a></center>''');
  run;

and, yea, I know the quoting is UGLY. I promise I'll blog about better ways to do this at some point in the future ;-).

and these ODS HTML statements generate the needed links in the HTML version.

  ods html text = &replayPDF
  ods html text = &replayRTF

And, as a bonus, since the ODS HTML statement does not write any content to the PDF or ODS destinations, those versions of the reports do not contain the hyperlink text!

And since the %stpEnd macro generates an

  ODS _ALL_ close;

statement, it will close all of the output destinations for you.

Feel free to check out this sample on my server.

And, of course, there are more examples of sessions and lightweight sessions in Building Web Applications with SAS/IntrNet®: A Guide to the Application Dispatcher. And they work pretty much the same way in the Stored Process Server (with the exception of the one issue highlighted in my PRIOR POSTING).

Happy Sessioning ;-).

NOTE: The blog of RTSL.eu: Project Plans in Excel - Grouping Tasks

2010-03-02T20:20:51+00:00

In the two previous posts in this series I described how to create a neat and simple Gantt chart and how to add dates to the day numbers. In this post I’ll describe how to groups your tasks in the chart that was featured in the previous post. The picture alongside (right) shows the end result from today's post. Grouping tasks is a generally useful thing to do, but I also find that my list of tasks increases as time goes by, so I might not need groups to begin with, but they become a useful way of keeping my plan tidy after it has grown.

As with the previous cases, I’m going to describe a quick and simple method. The objective is to have a useful and communicative chart without spending too long on creating it and without making it difficult to maintain. We start with the chart that was created in the last posting (shown to the left). Remember my comments in the first post in this series: I expect SAS developers to run their own (small to medium sized) projects from time-to-time, and I expect them to know how to work to a plan.

Let’s begin by adding a group column (inserted as column A, and then filled with appropriate values):

Select cells A2 to E6, i.e. the cells that have the column headers and our data. Launch the Subtotals window (in Excel 2002 it’s available from the Data menu). Use ticks to add subtotals to Start, Effort and End (in the summary rows we want to see summarised values for these three columns - yes, I know it's not appropriate to "sum" Start and End, just bear with me!):

When you click OK you’ll see that outlines have been added, and Start, Effort and End are being totalled (and sub-totalled). We need to change the “(sub-)totals” for start and end to show minimum and maximum instead (see, I asked you to bear with me!). Select column C and use Edit/Replace to replace “SUBTOTAL(9,” with “SUBTOTAL(5,”; and in column E, replace “SUBTOTAL(9,” with “SUBTOTAL(4,”. Finally, you might want to select column A and remove the word Total. The result looks like this:

Q: What was that stuff with changing subtotal 9 to subtotal 5 and 4?

A: Excel's subtotal function does more than subtotaling. The first parameter is a numeric code that tells the function what you want it to do. 9 indicates summarisation, i.e. adding-up, but 5 indicates you want to see the minimum value, and 4 indicates maximum value.

To finish our work we might a) Replace “Grand” in cell A9 with something more suitable such as “TOTAL”, and b) enhance our conditional formatting to make the (sub-)total rows look distinctive. We can spot them because the task name is blank in these rows. Thus we change the conditional formatting as shown below for cell G3 as an example (then just use Format Painter to spread it across the whole Gantt chart). Note the additional clause in the AND function of the first clause in addition to the extra condition:

And here’s the final result:

This series of three posts has shown how to quickly and easily create a simple Gantt chart. My objective has been to produce a neat, communicative chart without too much effort. In addition, the chart has to be robust and allow me to make changes (such as adding tasks or changing them) without disturbing the chart and requiring me to start from scratch. These charts demonstrate that the developer is in control of their project (without spending more time on planning than on delivering the goods).

If I'm working on a large enough project I sometimes add a couple of embellishments to the chart (given that the size of the project merits the extra effort involved). I like to highlight "today" and I like to track %done and show it on the pseudo-bars of the Gantt chart. I'll describe how to do this in a bonus post next week.

It is possible to use Excel charts to create a similar output. The chart can look more pretty than what we've achieved above, but I find that it can become unwieldy and difficult to read & print as the number of tasks grows. Hence, I choose to use the worksheet rather than a chart. I'll describe how to do this in a bonus post in a subsequent week.

NOTE: The blog of RTSL.eu: NOTE: Data Set Reader and Log Analyser from Oceanview

2010-03-02T20:08:17+00:00

UK-based Oceanview Consultancy have a couple of neat SAS utilities that just might meet your SAS team's needs. DSREAD reads a SAS data set and provides information about it without requiring SA software; and Elvis(!) is a log analyser par excellence.

Back in May 2006 when NOTE: was an email newsletter and boasted 3,700 subscribers worldwide, issue 17 featured the Elvis log analyser (and some awful usage of some of The King's record titles). Well, Chris Long wrote to me recently to tell me of his latest creation - DSREAD.

Chris describes DSREAD as a simple Windows command-line utility for working with data sets (SAS7BDAT ). The current version of DSREAD allows you to:

List the summary details of datasets
List the contents (the variable names and other attributes*) of datasets
Convert a dataset to CSV format for import into Excel and other software

The latter functionality could be useful in an automated system that uses a variety of technologies.

Chris lists a few limitations that you should read before downloading and using it. However, in a development environment it could prove useful.

It's good to see continued activity from Oceanview in the area of third-party SAS utilities (supported or otherwise). There are not many third-party SAS-related products available (the excellent Futrix being the notable exception).

With regard to Elvis, as issue 17 of the NOTE: email newsletter told us, it provides a novel and powerful environment for exploring your SAS logs. See the Elvis microsite for further detail.

Finally, if you like quizzes, you might like Chris's pet project: QuizBang. Visit the QuizBang web site, and see for yourself.

Key Happenings at support.sas.com: Referring to SAS and its products

2010-03-02T19:10:51+00:00

Q: How do I refer to SAS in a scientific paper?
A: Always write the name "SAS" in uppercase letters with no periods.

As you might guess, there are more guidelines and specifics about how to refer to SAS and its products and services. The complete list of guidelines is available on our corporate Web site under the Press Center in Editorial Guidelines.

I have selected a few items from the guidelines that may more accurately address the question that was intended.

Do not use a trademark symbol after the word SAS when referring to SAS Institute Inc.

Include the SAS trademark notice shown below in a footnote at the bottom of the page or the end of the article.

SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in the USA and other countries. ® indicates USA registration.

Scientific journals often require enough information to replicate your data findings. In these cases, the proper citation would be as follows (brackets "[]" indicate data that should be supplied by you):

The [output/code/data analysis] for this paper was generated using [SAS/STAT] software, Version [8] of the SAS System for [Unix]. Copyright © [year of copyright] SAS Institute Inc. SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc., Cary, NC, USA.

Remember that I have provided only a partial list. Visit the Editorial Guidelines for full details.

NOTE: The blog of RTSL.eu: NOTE: Don Henderson Has a Blog

2010-03-02T12:00:06+00:00

Somewhat humbly titled "Jurassic SAS in the BI/EBI World", long-time guru of SAS Don Henderson has joined the blogosphere. Don's CV is long and impressive and includes over 30+ plus years of using SAS;
Don is author of the SAS Press book "Building Web Applications with SAS/IntrNet: A Guide to the Application Dispatcher", and one of the principal architects for sasCommunity.org.

I've already learned useful stuff from Don's postings, including his January 26 post "An autoexec facility for the Stored Process Server" wherein he mentions that stored processes can have a program automatically run after each invocation (like a retrospective autoexec). I'd overlooked this useful feature.

JMP Blog: Credit Cards: A Method Behind the Madness

2010-03-01T14:00:00+00:00

It’s no surprise that a Google search for credit card offers yields more than 100 pages of links (and ads). Lou Valente, one of our design of experiments experts, recently used JMP to share a direct marketing DOE case study from the 2007 book Testing 1 - 2 - 3: Experimental Design with Applications in Marketing and Service Operations by Johannes Ledolter and Arthur Swersey.

A financial institution sought to sell more credit cards and to maximize return on new cards. Which fees and interest rates would yield the most attractive product to new customers? The institution considered offering products with a lower annual fee, a new fee for opening the account, a lower initial interest rate and/or a higher long-term interest rate.

They planned to mail 120,000 offers in the first pass and then to market the most attractive offer aggressively. Using JMP, Lou created a full 2⁴ factorial experiment to show how they identified the 16 combinations of fees and interest rates that they would offer. The institution outlined each offer in a separate invitation letter and sent each letter to 7500 prospects.

Which of the 120,000 offers got the highest response rates? Lou initially used simple Distributions in JMP to interactively investigate the relationship of the factors for the highest response rate. The institution learned that lower annual lower fee, no opening fee and a lower longer interest fee were associated with the highest number of offer acceptances. They were surprised to learn that initial interest was not as important.

Lou then built a model. Clearly those in the business of offering credit cards wanted to determine the highest acceptable annual fee and maximum initial interest the new customers could bear. Later, they will want to examine cancellations, delinquencies and profit from these new customers. Having that data would be delicious food for JMP junkies and could be helpful to each of us when considering new credit card offers.

Interested in more DOE? Join Lou's March 15 live Webcast, Design of Experiments to Solve Expensive Process Problems. Registration is required.

the SAS-BI blog: Quick Preview of Gauges and Additional Settings Available in BI Dashboard (4.2)

2010-03-01T09:00:13+00:00

Would you like to get a quick preview of the gauge you are using in the BI Dashboard Indicator? Just tap the 'Select ...' button to the right of the 'Gauge type:' dropdown box.

There are then images of what these gauges look like, such as the 'Dials' category below:

Also, if you select the 'Dynamic Gauges' category you will be able to edit the height/width of the gauge, display in 3D, the data format of the associated measure, and whether to orient the gauge horizontally or vertically:

SAS from Out in Left Field: SaviDataSet Alpha 1.0 On the Web

2010-03-01T07:24:17+00:00

I am finally ready with my SAS dataset reader/writer for .NET. It is written in 100% managed code using .NET 3.5. The dlls can be found
here.

A sample .NET console application can be found in the program file entries after the installation.

Update 2/13/2010:

I am now testing this against a real-world project so I am finding little bugs here and there. These are being addressed for version 1.0.0.1. If you need a build sooner than it is released, let me know.

I have also found that you need at around 100+ observations for this to work correctly. I am investigating but keep that in mind while testing.

[LATE BREAKING] I fixed the obs issue so I went ahead and uploaded the fixed version. Watch for breaking changes which are in the Readme.txt file.

Update 2/29/2010

Based upon Chris' work, I added in a console application that exposed my existing work. It allows for export of the sas7bdat to Excel, delimited, and XML. I will add in import at some point and have designed the interface as such. For example, I put the parms in an XML file to provide enough flexibility to accomplish everything. That has pros and cons but I figured the pros outweighed the cons.

I am also doing something I have meant to do for a long time which is to build a data viewer. Hence, I don't want an interim release but will bundle it all together. I have spent some time this weekend starting that process. All of the components are here but I have to pull them together which I am hoping to do this week.

SAS Programming: AUC calculation using Wilcoxon Rank Sum Test

2010-03-01T03:02:23+00:00

Accurately Calculate AUC (Area Under the Curve) in SAS for a binary classifier rank ordered data

In order to calculate AUC for a given SAS data set that is already rank ordered by a binary classifier (such as linear logistic regression), where we have the binary outcome Y and rank order measurement P_0 or P_1 (for class 0 and 1 respectively), we can use PROC NPAR1WAY to obtain Wilcoxon Rank Sum statistics and from there we are able to obtain accurate measurement of AUC for this given data.

The relationship between AUC and Wilcoxon Rank Sum test statistics is: AUC = (W-W0)/(N1*N0)+0.5 where N1 and N0 are the frequency of class 1 and 0, and W0 is the Expected Sum of Ranks under H0: Randomly ordered, and W is the Wilcoxon Rank Sums.

In one application example shown below, PROC LOGISTIC reports c=0.911960, while this method calculates it as AUC=0.9119491555


data test;
  do i = 1 to 10000;
     x = ranuni(1);
     if x + rannor(1) * 0.2 &gt; 0.7 then y = 1;
     else y = 0;
     output;
  end;
run;

ods select none;
ods output Association=Asso;
proc logistic data = test ;
    model y = x;
    score data = test out = predicted;
run;

ods select all;
data _null_;
    set Asso;
    if label2='c' then put nValue2=;
run;

%let Class=0;
ods select none;
ods output WilcoxonScores=WilcoxonScore;
ods output WilcoxonTest=WilcoxonTest;
proc npar1way wilcoxon data=predicted ;
     where y^=.;
     class y;
     var p_&Class;
     output out=wilcoxon;
run;

ods select all;
data AUC;
    set WilcoxonScore end=eof;
    array _A{2, 3} _temporary_;
    if Class=0 then do;
       _A[1,1]=N; _A[1, 2]=SumOfScores; _A[1, 3]=ExpectedSum;
    end;
    else do;
       _A[2,1]=N; _A[2, 2]=SumOfScores; _A[2, 3]=ExpectedSum;
    end;
    if eof then do;
       AUC=(_A[&Class+1, 2]-_A[&Class+1, 3])/(_A[1,1]*_A[2,1])+0.5;
       keep AUC; output;
    end;
run;

NPAR1WAY approach gets AUC=0.9119491555;
LOGISTIC reports c-statistic = 0.91160

SAS Programming: Partial Least Square

2010-03-01T03:01:28+00:00

In some predictive modelling projects, we may have variables that most of the observations have the same value, while the small percentage rest ones are populated with meaningful values. For example, 90% observations have values=0 but the rest 10% have value=1, 2, 3.....for a variable called WebHits, etc. We may have a large number of such variables, say web hits at different pages, and due to the small percentage of differently valued observations, each variable show minimal predictive power.

But we have a large number of such variables, and a quick way to figure out whether they collectively show up predictive power, we may use Partial Least Square method.

In SAS, we can get the PLS scores and score new data in this way:


ods select none;
ods output XWeights=Xweights;
ods output XVariableCenScale=XCenScale;
proc pls data=model_sample   method=SIMPLS  details censcale varscale;
        model  Event = &covars;
        
/* this is for comparison purpose*/
        output out=pls_tomvars(keep=PRSPCT_ID Enroll &covars _xscr_:) xscore=_xscr_;
run;
ods select all;


proc transpose data=XCenScale out=XCenScale_t;
        id variable;
run;

data XCenScale_t;
   if _n_=1 then _TYPE_='MEAN';
   else _TYPE_='STD';
   set XCenSCale_t;
run;

data XWeights;
       retain _TYPE_ 'PARMS';
      set Xweights;
      _NAME_=compress('_xscr_'||_n_);
run;


data Xscore/view=Xscore;
       set XCenScale_t XWeights;
run;

proc score data=model_sample  score=Xscore type=PARMS out=epsilon_std;
      var &covars;
run;

Lightweight yet detailed explanation of PLS and its application to data mining projects can be found at:
Pharmaceutical Statistics using SAS: A Practical Guide by Alex Dmitrienko, Christy Chuang-Stein, Ralph B. D'Agostino, SAS Publishing 2007

SAS Programming: SAS implementation of Kernel PCA

2010-03-01T03:01:16+00:00

Kernel method is a very useful technique in data mining that is applicable to any algorithms relying on inner product [1]. The key is applying appropriate kernel function to the inner product of original data space.

I show here SAS/STAT+BASE example code for Kernel PCA implementation. The example used is from Wikipedia @ here. Extention to other algorithms that suitable for Kernel method, such as CCA, ICA, LDA, etc, is straightforward.

Reference:
[1]. Zhu, Mu, "Kernels and Ensembles: Perspectives on Statistical Learning", The American Statistician. May 1, 2008, 62(2): 97-109

Figure 1. Linear PCA Result.

Figure 2. Result of Kernel PCA of Inner Product on quadratic polynomial formula

Figure 3. Result of Kernel PCA of Frobenius Norm on Gassian Kernel

SAS demo code:


/* Demonstrate kernel PCA. Kernel method can be applied to any algorithm that relies on inner product */
data original;
     do ID=-314 to 314;
        x1=sin(ID/100)*1+rannor(8976565)*0.1; x2=cos(ID/100)*1+rannor(92654782)*0.1; Class=1; output;
        x1=sin(ID/100)*2+rannor(8976565)*0.1; x2=cos(ID/100)*2+rannor(92654782)*0.1; Class=2; output;
        x1=sin(ID/100)*3+rannor(8976565)*0.1; x2=cos(ID/100)*3+rannor(92654782)*0.1; Class=3; output;
     end;
run;

/*------------ Linear PCA -----------*/
proc princomp data=original  standard noint cov  
              outstat=lin_stat(where=(_TYPE_ = 'USCORE'))
              noprint;
        var x1 x2;
run;


data lin_stat_score;
     set lin_stat;
     _TYPE_= 'PARMS';
run;

proc score data=original   score=lin_stat_score  type=parms   out=lin_pca;
     var x1 x2;
run;

proc gplot data=lin_pca;
        plot Prin1*Prin2=Class;
run;quit;


/*--------- Kernel PCA 1.0 ------------*/
/* Inner product based kernel PCA */
proc transpose data=original(drop=ID  Class) out=original_t; run;

proc corr data=original_t  outp=inner(where=(_TYPE_='SSCP' & substr(_NAME_, 1, 3)='COL')  drop=intercept)  sscp noprint;
     var col:;
run;

/* apply kernel functon to inner product matrix. Here we use (X'Y+1)^2, the one used in wikipedia.org example. */
data inner;
     set inner;
     array _C{*} _numeric_;
     _TYPE_='COV';
     do j=1 to dim(_C); _C[j]=(_C[j]+1)**2; end;
     drop j;
run;

proc princomp data=inner    noint  cov  standard 
              outstat=k_stat(where=(_TYPE_ in ('EIGENVAL', 'USCORE')) )
              noprint;
     var col:;
run;

/* get U by applying U=XV/S */
data S(drop=_NAME_)  score;
     set k_stat;
     if _TYPE_='EIGENVAL' then output S; else output score;
     drop _TYPE_;
run;

data _null_;
     set S;
     array _S{*} _numeric_;
     do j=1 to dim(_S);
        if _S[j]<0.5*constant('MACEPS') then do;
          call symput('maxsingval', j-1); stop;
        end;
     end;
run;
%put &maxsingval;

data score; set score; if _n_>&maxsingval then delete; retain _TYPE_ 'PARMS';  run;

proc score data=inner   score=score  type=parms  out=k_U(keep=Prin:);
        var col:;
run;

data k_U;
     set original(keep=ID Class  x1 x2)  ;
     set k_U;
run;

proc gplot data=k_U; 
     plot Prin1*Prin2=Class;
run;quit;



/*--------- Kernel PCA 2.0------------*/
/* Frobenius norm based kernel PCA */
proc transpose data=original(drop=ID  Class) out=original_t; run;

proc corr data=original_t  outp=inner(where=(_TYPE_='SSCP' & substr(_NAME_, 1, 3)='COL')  drop=intercept)  sscp noprint;
     var col:;
run;

proc means data=original_t noprint ;
         var col:;
         output out=_uss(drop=_TYPE_    _FREQ_)    uss= /autoname;
run;

proc transpose data=_uss  out=_usst; run;

proc sql noprint;
          select nobs into :NCOLS from sashelp.vtable
          where libname='WORK'
              and memtype='DATA'
              and memname='_USST'
              ;
quit;
%let ncols=%sysfunc(compress(&NCOLS));
%put &ncols;

/* apply kernel functon to inner product matrix. Here we use Gaussian kernel function. */
data inner;
          array _X{&ncols} COL1-COL&ncols;
          array _USS0{&ncols}  _temporary_;
          if _n_=1 then do;
             do j=1 to &ncols;
                  set  _usst(keep=COL1)  point=j;
                  _USS0[j]=COL1;
             end;
          end;
          set inner;
          _TYPE_='COV';
          do j=1 to &ncols;
              _X[j]=_USS0[_n_] + _USS0[j] - 2*_X[j];
              _X[j]=exp(-0.5*_X[j]/&sigma);
          end;
          keep _TYPE_  _NAME_ COL1-COL&ncols;
run;


proc princomp data=inner    noint  cov  standard 
              outstat=k_stat(where=(_TYPE_ in ('EIGENVAL', 'USCORE')) )
              noprint;
     var col:;
run;

/* get U by applying U=XV/S */
data S(drop=_NAME_)  score;
     set k_stat;
     if _TYPE_='EIGENVAL' then output S; else output score;
     drop _TYPE_;
run;

data _null_;
     set S;
     array _S{*} _numeric_;
     do j=1 to dim(_S);
        if _S[j]<0.5*constant('MACEPS') then do;
          call symput('maxsingval', j-1); stop;
        end;
     end;
run;
%put &maxsingval;

data score; set score; if _n_>&maxsingval then delete; retain _TYPE_ 'PARMS';  run;

proc score data=inner   score=score  type=parms  out=k_U(keep=Prin:);
        var col:;
run;

data k_U;
     set original(keep=ID Class  x1 x2)  ;
     set k_U;
run;

proc gplot data=k_U; 
     plot Prin1*Prin2=Class;
run;quit;

Further Reading:
Chapter 1 & 2 of
Principal Manifolds for Data Visualization and Dimension Reduction by A. Gorban, B. Kegl, D. Wunsch, A. Zinovyev (Eds.) Lecture Notes in Computational Science and Engineering, Springer 2007

StudySAS Blog: MedDRA and WHODrug Coding | StudySAS Blog [Digg]

2010-02-26T20:55:25+00:00

Though there are other Coding dictionaries available along with MedDRA (ex: COSTART and WHOART), MedDRA dictionary is typically used in the US for Adverse Events coding. AEMedDRA coding is not the SAS programmers work; it is usually done by medical coder or database programmer.

the SAS-BI blog: Improve Web Report Studio Performance by Carefully Choosing Processing Locations

2010-02-25T09:00:10+00:00

Information Map Studio allows data managers the ability to join tables, create new data elements, and label/format data items for business consumption. However due to this flexibility users must weigh the benefits of customizing information maps against the performance impacts of completing many of these tasks within the Information Map itself.

From anecdotal evidence (in 3.1), I have seen information maps successfully join small tables and display a few custom measures. However, when large tables are joined or a large number of custom measures are utilized, the performance (within Web Report Studio at least) can be dramatically impacted. (Note - I have found that running a test in Information Map for these cases is not indicative of the performance in Web Report Studio.)

So, once all the Web Reports are created and the performance improvements are required, how (in 3.1) can we switch where the custom measure is derived (from an expression within Information Map Studio to a data item in the source table/cube) without crashing the Web Reports?

Two options:

You can add these items and modify the associated web reports by hand - or
You can modify the XML to ensure that all the web reports use the new item instead.

Add the new measures into a copy of the original imap. Save out the xml & copy the new data element items from within the <ExternalResourceList ...> tagged area (located around the end of the xml file). Now in the original imap, export the XML and copy these new measures within the <ExternalResourceList ...> tagged area.

Then to switch the dataitem from a calculated expression to one of these new data elements follow the example below.

Included is what a simple calculated expression looks like in the information map XML:
<expr>
<resourceawarestrexpr exprtype="2" scope="2">
<reposobjuse prid="XR1">
<string strvalue="/">
<reposobjuse prid="XR2">
</resourceawarestrexpr>
</expr>

In order to 'trick' the information map into using a different data element, you need to change the entire xml block below into the following:
<Expr>
<ReposObjUse prid="XR3" />
</Expr>

Where the XR3 corresponds to the new data element item in the <ExternalResourceList ...> tagged area added earlier.

Of course - ALWAYS take a backup of the Information Map (and all associated WRS reports) since XML editing can introduce it's own problems.

NOTE: The blog of RTSL.eu: NOTE: Successes and Tools

2010-02-24T22:25:41+00:00

My previous post was our 100th and marks quite a milestone. Since starting in July last year, Google Feedburner tells us:

yesterday we achieved our highest ever number of subscribers: 156
we've clocked-up 36,000 views
our most popular post of all time is Dashboards & Scorecards: What To Measure with 1,055 views (note that the widget in our right-margin shows popularity over the last 7 days)

And Site Meter tells us that the average visit length is over 2 minutes, so we know that you're reading the articles as well as just looking at them ;)

The NOTE: blog was preceded by the NOTE: email newsletter, the archive of which is still available from our web site. As I reflected in my December What's It All About? post, this blog endeavours to cover the range of experiences and knowledge that a professional SAS developer might be expected to encompass. In other words, the blog includes hints and tips on SAS syntax, but it also covers a range of other topics related to software development including:

Requirements capture
Design decomposition
Modelling
Construction techniques and development processes
Refactoring
Configuration management
Peer review
Dynamic testing
Effort estimation
Written and verbal communication
Interpersonal skills

Thank you for supporting the blog, and I'm glad you find it of interest.

In doing my work for RTSL.eu I find myself away from the office a lot. To keep up the pace of posts I need a small collection of gadgets and technology. My favourites are:

My Acer Aspire One that's small enough and light enough to take into any pub and restaurant in the evening, yet with a big enough screen (and enough computing power) to make it eminently usable
The Blogger blogging platform and associated Google services (described in Building the Blog) that provide such a range of flexible means of maintaining the blog and its posts, and keeping track of its usage
Morose Media's WMWiFiRouter that allows me to remain connected whenever I need to be
My trusty HTC HD2 mobile phone, running Windows Mobile and Morose Media's WMWiFiRouter!

And finally, a special mention is due for DropBox. It's like a memory stick in the internet cloud, and it keeps all of my blog stuff synchronised between laptop, office computer and any web terminal I may happen to use. I highly recommend it, and the first 2GB are free! It uses Amazon's S3 storage service (so it's reliable), it adds its own encrypted AES-256 layer (so it's secure), and reviews that compare it with other services (such as JungleDisk) regularly rate DropBox as top due to its user-friendly integration with Windows, Mac and Unix (and iPhone, but not Windows Mobile).

If you're interested in DropBox, use this link to get a free account and contribute an extra 250MB to my own DropBox. Given that NOTE: is free and offers no advertising, I hope you won't mind me offering this referral link!

NOTE: The blog of RTSL.eu: NOTE: Success Demonstrates BI Scope (the BI evolution)

2010-02-24T21:45:51+00:00

Business Intelligence (BI) spans simple historic reporting to embedded real-time analytics. This is our 100th post and we're sharing our (minor) celebration with a SAS success.

BI is a commonly used term with a raft of different interpretations. Wikipedia begins to define it thus:

BI technologies provide historical, current, and predictive views of business operations. Common functions of Business Intelligence technologies are reporting, online analytical processing, analytics, data mining, business performance management, benchmarking, text mining, and predictive analytics.

There's a BI evolutionary path that starts with simple, static reporting on historic data (often delivered with spreadsheets) through to real-time predictive analytics embedded into front-office transactional systems. Many suppliers who claim to offer BI systems barely get off the ground on the BI flight to delivering real value to the enterprise.

All those products that offer sexy, shiny, slick graphics with animated 2.5D fuel gauges that make your historic data look exciting but don't begin to tell you about where you're headed are flattering to deceive. If you're considering implementing a BI solution, make sure your chosen software will give you the headroom to grow the value that the solution delivers. Don't box yourself in with a sexy solution that ultimately offers no real intelligence.

SAS salesmen will tell you that they can satisfy the full BI evolutionary path. And they'll point to their recent success in being awarded a Product of the Year award from Customer Interaction Solutions (CIS) magazine as evidence of offering solutions at the far end of the BI evolutionary path. In January, CIS magazine awarded a Product of the Year award to SAS Real-Time Decision Manager (RTDM).

SAS RTDM epitomises the ambition of true BI solutions. Incorporating business rules developed by the business users with a user-friendly graphical interface (and appropriate degrees of security), RTDM uses a service-oriented architecture to deliver SAS analytics into the heart of the customer interaction (telephone, web, ATM), promising recommendations, decisions and actions based upon the best information available.

The most popular reasons for considering a BI solution are to provide an analytical capability to more users, improve integration of data from a variety of sources, and to speed-up access to timely business data. Also, and by no means least amongst the reasons, businesses are often to keen to escape the bonds of spreadsheets and to gain a single, reliable & robust version of the truth.

In addition to the CIS magazine award, January saw SAS placed in the leaders' quadrant of the Gartner Magic Quadrant for Business Intelligence. Gartner's Magic Quadrant charts vendors' completeness of vision alongside those vendors' perceived ability to execute the vision. Gartner says that "leaders are vendors that are reasonably strong in the breadth and depth of their BI platform capabilities and can deliver on enterprisewide implementations that support a broad BI strategy."

Gartner's comments that are specific to SAS highlight SAS's focus on advanced analytics whilst noting that SAS customers use a broad range of SAS's BI capabilities. However, Gartner strikes a note of caution regarding the new challenges from big players like IBM and SAP who have recently bought previously-niche products such as SPSS, Cognos and Business Objects. Gartner suggest that SAS's comfortable position in the BI world may become less comfortable once the big boys get on top of their purchases.

I noted in December that SAS were aware of the changing market and felt up to the challenge. Given that SAS BI is currently in the Gartner leaders' quadrant and SAS Data Integration (DI) ranks equally highly in Gartner's view, SAS will consider themselves facing the challenge from a position of strength.

NOTE: The blog of RTSL.eu: NOTE: Issue 49 of VIEWS News is Available

2010-02-24T12:05:00+00:00

Phil Holland has published issue 49 of VIEWS News, the quarterly journal of the VIEWS International SAS Programming Community. This quarter's issue includes graphical impact, encryption algorithms & format merges, and Excel functions available in SAS 9.2, plus a prize draw especially for VIEWS News readers.

You can view the 48 back-issues in the archive.

NOTE: The blog of RTSL.eu: NOTE: PROC MEANS Gives You All It's Got (and More!)

2010-02-24T12:00:00+00:00

An oft overlooked parameter for PROC MEANS (and PROC SUMMARY) is COMPLETETYPES. It tells MEANS to create all possible combinations of the values of the classification variables, even if some of those combinations don't exist in the data. And PRELOADFMT will create combinations from values that don't even exist in your input data. This can be very useful in presenting what appears to be a more complete picture of the input data and can be equally useful in presenting a consistent layout amongst a group of reports (or regularly produced reports).

Here's a simple code example:

data sales;
  region = 'North'; product = 'Widget'; sales = 500; OUTPUT;
  region = 'North'; product = 'Foobar'; sales = 300; OUTPUT;
  region = 'South'; product = 'Widget'; sales = 100; OUTPUT;
run;

proc means data=sales /*completetypes*/ sum;
  class region product;
run;

If we run it without COMPLETETYPES we get:

     Analysis Variable : sales

region    product    Obs             Sum
----------------------------------------
North     Foobar       1     300.0000000
   Widget       1     500.0000000
South     Widget       1     100.0000000
----------------------------------------

And if we run it with COMPLETETYPES, we get (with the new information highlighted in red):

     Analysis Variable : sales

region    product    Obs             Sum
----------------------------------------
North     Foobar       1     300.0000000
   Widget       1     500.0000000
South     Foobar       0               .
   Widget       1     100.0000000
----------------------------------------

If there's a chance that your data won't contain all possible values for any given classification variable (this is a possibility when you regularly run reports on differing data), you might want to consider PRELOADFMT - it's a key word for the CLASS statement. The formats associated with the CLASS variables are used to tell MEANS the full list of values. See this example (key code highlighted in red):

proc format;
  value $regions 'North' = 'North'
   'South' = 'South'
   'West' = 'West'
   ;
run;

data sales;
  format region $regions.;
  region = 'North'; product = 'Widget'; sales = 500; OUTPUT;
  region = 'North'; product = 'Foobar'; sales = 300; OUTPUT;
  region = 'South'; product = 'Widget'; sales = 100; OUTPUT;
run;

proc means data=sales completetypes sum ;
  class region product / preloadfmt;
run;

We define a format for REGION which contains "West", but West is not in our data at all. We add PRELOADFMT as an option to the CLASS statement, and here's the output that we get (new bits in red):

     Analysis Variable : sales

region    product    Obs             Sum
----------------------------------------
North     Foobar       1     300.0000000
   Widget       1     500.0000000
South     Foobar       0               .
   Widget       1     100.0000000
West Foobar       0               .
   Widget 0               .
----------------------------------------

So, we've seen how we can zero-fill a data set using two different techniques, the first using all combinations of data within the input data set, and the second using a pattern of values from format(s).

See also:

PROC MEANS - CLASSDATA, MISSING
PROC FREQ - SPARSE

Sadly, the Summary Statistics task in Enterprise Guide (which uses PROC MEANS) does not offer COMPLETETYPES nor PRELOADFMT; but you can add them yourself within the Code Preview window.

NOTE: The blog of RTSL.eu: NOTE: More Online Training

2010-02-24T11:35:00+00:00

Having mentioned Sunil's online training and featured his mooted courses in India yesterday, I have to mention Andrew Karp's ever growing catalog of online and face-to-face courses at Sierra Information Services. And the Virtual SAS Users Group who offer yet more online courses. Sitting alongside the online courses that SAS offer, there's a wealth of material waiting for you to sign-up and learn more about the technology and your industry.

JMP Blog: Creating a Simple Dashboard in JMP

2010-02-23T18:43:37+00:00

Business users have been flocking to JMP for its data visualization and analytical capabilities. You can create a simple dashboard with a minimal amount of scripting. The functions that I will use are: New Window(), H List Box(), and V List Box().

The New Window() command creates a new report window where all of your analyses will be placed. H List Box () and V List Box() allow you to visually arrange analyses horizontally and vertically, respectively. Without specifying H List Box() and/or V List Box(), all analyses will appear stacked on top of each other.

Let’s build a dashboard step by step:

1. In a new script window, use the following code to open a JMP data table.

 MyDataTable = Open( "$SAMPLE_DATA/Big Class.jmp" );

I am using the Big Class data table found in the JMP sample data directory. However, your path may look something like this: “C:/My JMP Analyses/Favorite JMP Data.jmp”.

2. Create a new report window.

MyDataTable = Open( "$SAMPLE_DATA/Big Class.jmp" );

New Window( "Student Report",  );

The New Window() command requires a title, in this case “Student Report,” and a display box which is left empty for now.

3. Add an analysis to the report. I recommend using the point-and-click method to run an analysis and then take the JSL code that was written behind the scenes. Under the Hot Spot (red triangle) of your analysis report, select Script and then Copy Script. Add the script to your New Window() command as the display box. I’ve used the Bivariate Analysis.

MyDataTable = Open( "$SAMPLE_DATA/Big Class.jmp" );

New Window( "Student Report", Bivariate( Y( :weight ), X( :height ) ) );

4. Add additional analyses by making sure to end each analysis script with a semi-colon. I’ve added Tabulate.

MyDataTable = Open( "$SAMPLE_DATA/Big Class.jmp" );

New Window( "Student Report",  

	

	Bivariate( Y( :weight ), X( :height ) );

	Tabulate(

		Show Control Panel( 0 ),

		Add Table(

			Column Table( Analysis Columns( :height, :weight ), Statistics( Mean ) ),

			Row Table( Grouping Columns( :sex, :age ) )

		)

	);

	

);

5. String together analyses horizontally by using the H List Box() command. Display boxes are strung together using a comma: H List Box(Display Box 1, Display Box 2,…). Similarly, string analyses vertically by using V List Box(Display Box 1, Display Box 2,…);

 MyDataTable = Open( "$SAMPLE_DATA/Big Class.jmp" );

	

New Window( "Student Report",  



	H List Box(

	

		Bivariate( Y( :weight ), X( :height ) );

		

		Tabulate(

			Show Control Panel( 0 ),

			Add Table(

				Column Table( Analysis Columns( :height, :weight ), Statistics( Mean ) ),

				Row Table( Grouping Columns( :sex, :age ) )

			)

		);

	)

	

);

6. You'll find a sample dashboard with multiple analyses stacked horizontally and vertically in the JMP File Exchange under my name. Further dashboard customization, such as coloring data points and adding a title, are also found in the example.

the SAS-BI blog: The easiest way to generate additional log information for Web Report Studio (3.1 and 4.2)

2010-02-23T13:34:28+00:00

Ran across a situation last week that proved extremely helpful. We needed to generate detailed logs on Web Report Studio use in order to verify pooling was setup correctly and then to monitor performance of some poor performers. The easiest and most straightforward way to accomplish this was to follow these steps.

1. Verify that the Tester is a member of the WRS Administrator group in SAS Management Console
2. Log into SAS Web Report Studio
3. Modify the URL to: /SASWebReportStudio/debugLog.do?LogAllActions=true and hit 'Enter'
4. You can also change the debug level by modifying the URL to the following format

/SASWebReportStudio/debugLog.do?LogName=log.context&LogLevel=level

An example:

/SASWebReportStudio/debugLog.do?LogName=com.sas.apps.citation&LogLevel=DEBUG

and hit 'Enter'
5. Continue with testing.
6. Don't forget to turn off logging before getting out of Web Report Studio

/SASWebReportStudio/debugLog.do?LogAllActions=false

7. Log off of WRS and retrieve the WebReportStudio.log file

For 3.1 - the reference material is available on pg 110 of http://support.sas.com/documentation/configuration/biwaag.pdf

For 4.2 - the same instructions apply. See the section 'Configure Debug Logging Dynamically' on page http://support.sas.com/documentation/cdl/en/biwaag/63059/HTML/default/a003050065.htm.

NOTE: The blog of RTSL.eu: What is a Project?

2010-02-23T13:18:17+00:00

Alongside our series of posts on creating Gantt charts in Excel for the purpose of managing small to medium sized projects, a discussion on "what is a project?" might be useful. Most of us feel we understand the general usage of the term "project", but what does it mean in the context of Project Management?

The Cambridge Advanced Leaner's Dictionary defines the noun project as "a piece of planned work or an activity which is finished over a period of time and intended to achieve a particular aim". The key attributes are a) having an aim or purpose, and b) being able to define a start and end date/time. These attributes make projects distinct from ongoing operational, live, production or business as usual (BAU) activities.

The existence of the end date is an important element of the project, and a lot of work goes into agreeing the end date and then making sure the project is delivered/completed by the end date. Plans are drawn-up, often including Gantt charts.

Projects exist to create something new; something that hasn't been created before. So, we can have a project to create a new business intelligence solution for a new SAS customer, or we can have a project to create a new data mart. Producing a bunch of standard reports every week is not a project, but the task of producing the reports for an individual week might be considered a project because it has a defined aim and a defined end date.

An important element of project planning is to be sure that everybody understands what "finished" means, and what it looks like. Does your development project finish when your code is successfully tested and goes live, or will their be a "warranty period" wherein the development team will be on-hand to investigate and resolve problems quickly, or does all of the user training need to be delivered too?

It is the responsibility of everybody involved with the project to make themselves aware of the full set of deliverables and the dates by which they should be delivered. And it is every team member's responsibility to focus on achieving these deliveries on-time, on-budget, and with the expected degree of quality.

I should add that sometimes projects finish before they have delivered their aims. It is good practice to monitor progress throughout the life of the project, but it's also good practice to review whether the aim is still achievable and whether the expected benefits will still accrue as a result. If you have a multi-year project aimed at producing a new, improved widget yet (since you started your project) the world has discovered that foobars are preferable and everybody's getting rid of their widgets, you better give serious consideration to canning your project (or resetting its aims, or increasing the marketing budget for your widgets!).

NOTE: The blog of RTSL.eu: Project Plans in Excel - Adding Dates

2010-02-23T13:17:03+00:00

In the previous post in this series I described how to use Conditional Formatting to create a neat and simple Gantt chart alongside a simple Excel-based project plan. In this post I’ll describe how to use dates in addition to the day numbers that were featured in the previous post. The picture alongside (right) shows the result from today's post.

As with the previous case, I’m going to describe a quick and simple method. This method also takes weekends into account as non-working days. We ended the last post with what you see alongside (left).

So, let’s begin by adding the date for day 1 into cell F1. I'm typing “22/2” to represent 22nd February). It’s not readable in the small width of the cell, so we’ll go to the Format Cells window (you can use Ctrl-1 to get there quickly) and select text orientation as 90 degrees. Then, to get the date format that we want, we’ll stay in the Format Cells window and specify a custom number format of “dd-mmm (ddd)”. If the height of row 1 doesn’t automatically increase for you, just do it manually. You should have a result like this:

Now, let’s set G2 as =F1+1 and copy the formatting from F1. This gives us the correct result for G2, but it’s not a sequence that we can ask Excel to follow because we’ll get weekends included, so we should set the formula for G2 to =IF(WEEKDAY(F1)<>6,F1+1,F1+3). In other words, if the previous day is not a Friday, just add 1 to the date; if it is Friday, add three to the date. We can copy and paste G2 across the remainder of the row:

So, with very little effort, we have a Gantt chart with day numbers and dates.

In one further posting I’ll describe how to add groups of tasks and how to summarise them.

NOTE: The blog of RTSL.eu: Project Plans in Excel (simple, quick and effective)

2010-02-23T13:12:17+00:00

For any piece of work other than the smallest, it’s worth planning. Planning doesn’t have to mean creating a huge monster in Microsoft Project - I find that Microsoft Excel (or similar) is often sufficient (and a lot more accessible to the team). This post (and the series of posts that follow) describes how to quickly and efficiently create an adequate plan for small to medium sized projects.

I don't expect all developers to be expert project managers, but I do expect my team members to understand the role of the project manager, to know how to work to a plan, and to focus on delivery. And I do expect developers to run their own (small to medium sized) projects from time-to-time.

A project plan can consist of just a list of tasks (preferably with start and end dates) together with the name of the person who will complete the task, but this can be made to communicate a lot more if you can deliver a Gantt Chart too. The name “Gantt Chart” sounds challenging to anybody who hasn’t met one before, but actually it’s rather simple format that most people are familiar with (often without knowing the name). Gantt Charts can contain a lot of detail and embellishment, but I’m going to describe how to create a simple yet communicative chart very quickly.

To illustrate what to do, I’ll use Microsoft Excel 2002, conditional formatting, and formulae.

Let’s start by creating the basic list of tasks, their start dates, and their durations (I’ll ignore the name of the actionee in order to save space):

We can add a formula to calculate the end date: d3 set to =B3+C3-1, etc.

Next, let’s add columns for the days of our Gantt:

To create our Gantt, we want to create solid-coloured cells where the day number (in row 2) lies between or equal to the start and end days respectively. To do this we will use Conditional Formatting. From cell F3 we need the following Conditional Formatting (accessed from the Format menu in Excel 2002):

In addition, we’ve used the Format button to set the pattern for the cell shading to solid green. And, having clicked OK, we see that cell F3 turns green, reflecting the fact that activity #1 is to be performed on day 1.

We now need to copy this formatting to all of the other cells of the Gantt, but if we copy and paste we’ll get the wrong results because the cell references will be move relative to the point where we paste the formatting. So, we need to change the formula to tell Excel which bits of the formula should move when we copy/paste, and which bits should be fixed. The modified formula is: =IF(AND($B3<=F$2,F$2<=$D3),1,0)

Armed with the correct formula, we can use the format painter to copy the formatting, or we can use Copy and then Paste Special. Either way, having done so we will see the end result:

That’s a simple, quick, but communicative Gantt chart. In subsequent posts I’ll describe how to quickly translate day numbers into dates, how to add grouping of tasks, and how to indicate progress (% done).

What I have described is just one of many routes to the same end. There are plenty of resources on the web to guide you to a similar result (or better) with Microsoft software and with others. Plus, SAS/OR offers plenty of project management functionality, including Gantt charts from PROC GANTT.

NOTE: The blog of RTSL.eu: NOTE: New Course in India: Best Practices in SAS Statistical Programming for Regulatory Submission

2010-02-23T11:30:00+00:00

Having successfully conducted his long-titled course in the USA, Netherlands, and online, my friend Sunil Gupta is considering the possibility of running it in one or two cities in India. While the schedule is not yet finalised, the favoured period for the two-day class is around April end/early May in Bangalore and maybe Hyderabad.

For further details of the course, see the overview of the online version. To register interest in the Indian course(s), email Sunil directly (and tell him NOTE: sent you!).

You might also like to know that Sunil is presenting his "Preparing SAS Programmers for the Pharmaceutical Industry (An Introduction)" course as a pre-conference course at SAS Global Forum (SGF) 2010. There may still be places available.

The Princess of Science: ‫The Simpson‬

2010-02-23T10:09:04+00:00

‫במשחק השביעי והמכריע בסדרת גמר אליפות הכדורסל הארצית, התמודדה קבוצת ספרינגפילד בולס מול יריבתה המושבעת, קבוצת יוטה סופרגז. כצפוי, המשחק הוכרע על פי היכולות האישיות של כוכבי שתי הקבוצות: בארט מספרינגפילד ויוחנן מיוטה. במחצית הראשונה היו לבארט 40% אחוזי קליעה מהשדה, בעוד שיוחנן צלף ב- 50% מנסיונות הקליעה שלו. במחצית השניה צפינו בהתעלות אישית של [...]‬

Key Happenings at support.sas.com: SAS gets your attention with a text message!

2010-02-22T19:01:42+00:00

Contributed by Scott Vodicka, a member of the SAS Global Consulting Business Intelligence Practice

An interesting topic came up the other day in one of my email conversations. What was the topic of the email? Well, I am glad you asked. It was how do we send text messages from SAS to someone's cell phone?

It is so easy that you will not even believe it! As a lot of us know SAS supports sending emails from a data step program, and the publish and subscribe model supports delivering information via email. The quick answer is to use the email address of the person's phone in the TO: field for the data step, or the email address used to subscribe to a channel.

So, now that the easy part is done, let me give you some information that will help you solve the problem I know you are now facing: How do I determine the users email address for their phone? There are a couple of tricks that make this really easy.

Below is a short list of popular carriers with their associated domains:
  AT&T:          @txt.att.net
  Alltel:           @message.alltel.com
  Sprint:           @messaging.sprintpcs.com
  T-Mobile:         @tmomail.net
  Verizon:          @vtext.com
  Virgin Mobile:     @vmobl.com

Check out these web sites for updated information on carriers, especially since this sector is constantly in flux.

http://en.wikipedia.org/wiki/List_of_carriers_providing_SMS_transit

http://edgedirector.com/htm/page.htm

If your phone number is: (919) 555-1234 (not a valid number), and your carrier is AT&T then the email address for your phone is: 9195551234@txt.att.net

Another way to find out your phone's address is to send a text message from your phone to your SAS email account: and you’ll get the email address for your phone in the from field (it might go to your junk folder).

Here is a sample data step program that sends an alert as a text message to a user's cell phone. Before you submit the program below be sure to update your SAS configuration file for the following options:
EMAILSYS EMAILHOST EMAILID

You may have to supply the EMAILPW value depending on how your SMTP server is configured. You may specify the EMAILID and EMAILPW values in your SAS program via an options statement. See SAS documentation for details.

%let email_alert="9195551234@txt.att.net";

%let email_from="SAS Monitoring System ";



%let errormsg=Server failed to start;

%let host=metadata_server;



filename em_out email to=(&email_alert) from=(&email_from) 

    subject="Alert: SAS 9.2 Critical Status Error";

data _null_;

  file em_out;

  format timechar $21.;

  timechar=put(datetime(),datetime21.2);

  put timechar "ERROR : &sysuserid.@&host - &errormsg..";

run;

Important Note: Many people are charged per text message. Be sure to get permission before sending a text message to an individual.

SAS Global Forum: “Enormous Power at my Fingertips”

2010-02-22T14:43:00+00:00

Last month I pointed you towards the conference t-shirt contest on sasCommunity.org. The designs and coding techniques on display were quite creative, and the contest triggered some fun conversation among community members. Now the votes have been cast.

And the winner is …self-proclaimed “new kid” Lynne Krajevski. I loved what she had to say about her unique design: “SAS is enormous power at my fingertips, always within my reach - this is what I was trying to convey with my design.” Way to go Lynne!

How can you get one of these limited number t-shirts? Come to SAS Global Forum 2010 in Seattle to find out!

JMP Blog: Saving Graphs, Tables and Reports in JMP

2010-02-22T14:24:00+00:00

Here are some simple ways to save both tabular output and/or graphs in JMP.

When you copy information from JMP, you can either use “Copy” from the Edit menu or the tool bar. The copied information is copied to an active windows clip board.

1. If you want to save the whole report output with graphs, simply choose Edit/Copy to store the active window to the clipboard and then paste the results in your document.

2. You can save just a portion of the results by choosing the “fat-plus" tool from the toolbar (or Tools>Selection) and then clicking on the table, column or graph you would like save. You then copy and paste as before.

a. From any JMP output window, click on the selection tool in the toolbar (the fat plus)
b. Click on the content you’d like to copy – the selected content is highlighted. To extend a selection, hold the Shift key
c. Click Edit>Copy

d. Open the program where you’d like to paste the content and select Paste>Paste Special. From the list, select Picture (Enhanced Metafile).

By selecting Picture (Enhanced Metafile), you have the ability to grow and shrink the image, ungroup the image for manipulation. This options also takes up less storage space.

Once the image has been pasted in your application, if you right-click within the image and select Edit Picture, you then have the ability to manipulate that image.

3. JMP also provides methods that allow you to customize report output and save it as a graphic, text, Word, .rtf, or .html. The text format is the only format that doesn't save graphs. Select File>Save as

4. You can also save selected graphs or tables by choosing Edit>Save Selection As. This option only allows you to save the selected item as graphics with an extension .png, .jpg, .svg, .gif or .wmf

5. You cannot save output directly from the report; instead, you must duplicate it by selecting Edit>Journal or Edit>Layout. Both commands will reproduce the report in a new window and allow you to customize the look of the report. The layout option provides a few more editing options, such as allowing you to ungroup parts of a report. Select File/Save and choose a format above to save a journal or layout window.

To learn more about saving data tables, graphics and reports, refer to Chapter 4 of the User Guide in JMP: “Saving Data Reports and Sessions.”

The SAS Dummy: Is your database on your platform supported by your version of SAS?

2010-02-22T14:06:09+00:00

Which versions of MySQL are supported by SAS 9.1.3 for Windows? Is the 64-bit version of Oracle 11g supported on the 64-bit version of SAS 9.2 on Linux? You can find the answers to these questions and others like it by consulting the SAS/ACCESS Validation Matrix: pick your SAS version, your database, and your architecture and -- BAM! -- get your answer in no time flat.

NOTE: The blog of RTSL.eu: NOTE: Enterprise Guide Add-Ins Are Out There

2010-02-19T13:09:41+00:00

I wrote recently about the paucity of 3rd-party add-ins for Enterprise Guide. I finished the post by wondering aloud whether there were more that I hadn't found. Well, I've had no reports of any 3rd-party add-ins, but in Chris Hemedinger's SAS Dummy blog he summarised a good range of SAS-supplied add-ins that are available for free download.

Whilst they are largely unsupported, they certainly provide some jolly useful functionality. And the source code is provided so you can support them yourself.

Thanks Chris!

NOTE: The blog of RTSL.eu: NOTE: Informats and Enterprise Guide 4.2 (beware)

2010-02-19T13:09:16+00:00

I'm a keen follower of Susan Slaughter's books (in conjunction with Lora Delwiche) and her Avocet Solutions web site. The web site is very nicely structured and contains a wealth of solid information. Last week, the Little SAS Book Series featured an article about informats and Enterprise Guide 4.2. The article highlighted useful, user-friendly features of EG 4.2's Data Grid, but also warned of the fact that said Data Grid ignores informats.

The site and the article are recommended reading.

NOTE: The blog of RTSL.eu: Encourage the (New) Conference Speakers

2010-02-19T09:49:39+00:00

Reflecting further on the unconference & BarCamp format of the Analytics Camp NC event that I mentioned last week, whereby sessions are proposed and scheduled each day by the attendees and based upon pitches from the potential speakers, I realised that this is a good means of giving feedback to potential speakers and thereby encouraging new speakers.

It's a little daunting to write a paper and send it off to some anonymous conference organiser in the hope that you might be seen to offer something of interest to fellow conference attendees. And I've recently seen at first-hand how conference organisers can be dismissive of those whose papers are not selected (to the extent of not even bothering to tell them that their paper was not selected). To get some constructive criticism out of them, in order to do a better job next time around, can be like getting blood out of a stone. People who have had papers accepted for conferences on previous occasions will not be put off by such behaviour; however, for a first-timer the anonymous rejection can easily put them off of ever submitting a paper again.

By contrast, the atmosphere at Analytics Camp seems to have been very informal and welcoming. It sounds like just the sort of atmosphere where a novice might be tempted to propose a topic and be given positive encouragement to proceed with their idea.

I continue to warm to the unconference & BarCamp ideas and ideals. More importantly, if you're organising a conference, please be sure your section chairs show respect and offer encouragement for all of those who take the time and effort to prepare a paper and submit it. For conferences to thrive they need a regular influx of new thoughts and ideas; don't stifle and discourage first-timers.

The Princess of Science: ‫Take Me Out To The Ball Game‬

2010-02-19T08:41:08+00:00

‫השיר Take Me Out To The Ball Game מסכם את כל חווית הבייסבול. כשלקחתי את בני הבכור למשחק לראשונה בחייו (קאבס נגד הקרדינלס, הפסדנו 8:5 למרות ההום ראן ה-7 מתוך 66 שסמי סוסה הכה באותה עונה) הוא אמר לי בדרך:אני מקווה שלא יהיה shame, מתייחס בבירור לשורה מהשיר ""If they don't win it's a [...]‬

The SAS Dummy: A blog by any other name would smell just as...well, nevermind

2010-02-18T13:57:00+00:00

Within SAS, we have a strong blogging community made up of SAS employees. Those of us who read or contribute to the blog content on the internal SAS web got a special "shout out" from Fortune magazine, within its coverage of SAS as the top workplace in the USA.

The print version of the magazine includes a sidebar titled, "An Army of Bloggers, 600 Strong", which discusses the SAS intranet with its cafe menus and work-life announcements, and especially the bloggers. Here's an excerpt:

More than 600 employees have their own internal blogs. They're typically used to talk shop. Chris Hemedinger writes the "Smells Like a Log" blog, which covers, among other things, unusual uses for the SAS programming language. One post discusses how SAS helped Kogi State University study "the effect of nut storage, nut size, and nut-soaking on nut-sprouting."

I'm accustomed to people thinking that I have a funny-sounding name, but I've never felt so conscious about the name of my internal-only SAS blog. Now that the blog title has appeared in a national publication, I feel like I've got some explaining to do.

I began keeping my internal SAS blog over 5 years ago. Blogs were definitely not a new thing, but it was still in an era when some people called them "weblogs". So it's a log, but on the web. Where does the smell come in?

I'm a software developer (okay, a manager who sometimes poses as a coder). Several years ago I wrote a code routine to examine the contents of a text file and determine whether it contained log output from a SAS program. What did I call the routine? SmellsLikeALog(). (It returns True or False, depending on whether the text file contained the markers for SAS log output.)

My colleagues thought that was a funny name, because (as any 4th grader will tell you) "smells like a log" has some hilarious connotations that have nothing to do with programming.

When I founded my internal blog and had to assign a title, "Smells like a log" was the first (and only) thing that came to mind. Blogs were a bit new at SAS, and I didn't think the title was going to have much significance. Who knew?

Now that I've told the story behind the title, I'm not convinced that the explanation will really boost my personal brand. But it's the truth, and I know that if you don't tell the truth to the media, the bloggers will ferret you out, expose you, and you'll wind up smelling...well...less than rosy.

Jurassic SAS in the BI/EBI World: Sessions - A double edged sword

2010-02-18T08:45:44+00:00

In my previous post (which was longer ago than I'd like - but maybe I can blame the US East Coast Blizzards) I blogged about using Sessions as a surrogate for some of the things we old-style SAS programmers used the WORK library for.

One of the things that many of us overlook when delivering SAS content over the web is that each request has to be quick - users get very impatient if they have to wait too long. So long-running programs with lots of steps should not just be packaged up as Stored Processes. They need to be broken down into pieces/parts that can be run sequentially - just like one can run a series of DATA/PROC steps in an interactive SAS session. And that is where sessions can come into play - we can use a session to save data sets (and macro variables) from one request to the next). So the Sessions SAVE library becomes the replacement/alternative for WORK.

So here goes with a little tutorial on creating a Stored Process that uses sessions. Let me begin by saying that I will of course be packaging this as a macro so I can leverage my runMacro Stored Process.

The following macro code shows a simple template for how to use sessions. Note how the sysfunc macro is used to invoke the libref function which checks if the SAVE library exists. Since the SAVE library is created when the sesssion is created and is made available when reconnecting to a session, that is a convenient way to check if we already have a session.

%macro sessionsExample;
 %global save_Session_Created;
 %if %sysfunc(libref(save)) ne 0 %then
 %do;  /* session does not exist - create it */

 %end;  /* session does not exist - create it */
 %else
 %do;  /* reconnecting to the session */

 %end; /* reconnecting to the session */
%mend sessionsExample;

In the first %do-%end block we add some code to create a session:

 %let rc = %sysfunc(stpsrv_session(create));

and create a macro variable to be saved (note the prefix save_)

 %let save_Session_Created = %sysfunc(datetime(),datetime.);

and, in this case some sample code to perhaps access a large table and summarize it - so we only have to do that once (and please forgive me for cheating :-) here and using a small SASHELP data set).

proc summary data = sashelp.shoes nway;
 class product;
 var stores sales returns inventory;
 output out=save.SummaryTable(drop=_freq_ _type_) sum=;
run;

We then list the data and use a title to generate a hyperlink to reconnect to the session.

proc print data = save.SummaryTable;
 title "Session Created at &save_Session_Created";
 title2 '<a href="' "&_thisSession" '&_program=' "&_program" '&macroToRun=' "&macroTorun" '">Reconnect</a>';
run;

The macro variable _thisSession is the key here - this is what causes the request to connect back to the same session as described in this SAS 9.1.3 Doc. (and it works pretty much the same in 9.2)

And in the code where we are reconnecting, we can access the macro variables and the data we saved in the SAVE library, e.g.,

proc report data = save.SummaryTable;
 columns product stores sales returns inventory;
 rbreak after / summarize;
 title "Reconnecting at %sysfunc(datetime(),datetime.) to Session Created at &save_Session_Created";
 footnote '<a href="' "&_thisSession" '&_program=' "&_program" '&macroToRun=' "&macroTorun" '">Reconnect Again</a>
run;

Feel free to try it on on my server.

There are lots of other example of sessions described in my SAS Press Book. They work pretty much the same way in the SAS 9 Stored Process server with a few minor differences described on this page about upgrading SAS/IntrNet program to Stored Processes (the most notable is the format change for the _REPLAY macro variable).

So by now, if you made it this far :-), you are probably confused by the reference to the double edged sword. Check out the limitations at the bottom of this page - overuse of sessions can seriously impact performance. So only use em when you need em!

StudySAS Blog: SAS Tip: Assigning a flag with value 1 or 0 to variable [Digg]

2010-02-18T01:07:37+00:00

*Creating a flag variable when a test variable meets certain criteria is very common for SAS programmers….

SAS Programming: Clustering Handwirtten Digits (digitalized optical images)

2010-02-17T18:43:35+00:00

In this example, we show the a clustering exercise on Optical Recognition of Handwritten Digits Data Set available @ UCI data set repository (Link).

This exercise is a standard application of HOSVD by stacking 8X8 matrix of digitalized bitmap of each letter where 1-mode is digits for each column and 2-mode corresponds to rows while 3-mode is each letter object, a standard Tensor representation. U_(3) from HOSVD of this Tensor gives indication information for clustering. In this exercise, we apply k-means algorithm on the correlation matrix of each row of U_(3), which corresponds to each subject, with each column of right eigenvector matrix to eliminate outlier effect and stablize the matrix for final clustering algorithm.

The clustering result, together with original unclustered correlation data and correlation data sorted by true letter ID, is shown below. In order to facilitate display, we show the confusion matrix of the correlation matrix so that the clustering result is more obvious to identify.

Cross Tabulation of True Class (0-9) and Clustering Result (1-10):

Further thoughts. The confusion matrix display shows that while the letter clusters have shown up but the pattern is not as prominent as we expected. However, it is possible to obtain better result by applying Nonnegative Matrix Factorization (NMF) to the square of correlation confusion matrix. The NMF algorithm will be implemented in SAS soon.

Note: All numerical analyses are done using the prototype code demonstrated at this Blog Entry (Link). This figure is drawn in R.

【 Clustering Result shown in Confusion Matrix style 】

Reference:
Matrix Methods in Data Mining and Pattern Recognition by Lars Eldén

SAS Programming: Implementing Gap statistic for clustering number estimation

2010-02-17T18:43:11+00:00

Gap statistic is a method used to estimate the most possible number of clusters in a partition clustering, noticeablly k-means clustering. This measurement was originated by Trevor Hastie, Robert Tibshirani, and Guenther Walther, all from Standford University.

The final estimate of cluster number is
min_K Gap(K)>=Gap(K+1)-s(K+1)

Code below implements uniform sampling approach (method (a) in their paper, samplemethod=1)and SVD sampling approach (method (b), samplemethod=2).

The simulated data has 5 clusters as seen below, while the Gap Statistic correctly identifies this number of clusters as:
\hat_k = 4, Gap=1.37285, Gap_(K+1)-s_(K+1)=1.6494
\hat_k = 5, Gap=1.65057, Gap_(K+1)-s_(K+1)=1.6485, Gap_K>[Gap-s]_(K+1)
\hat_k = 6, Gap=1.64965, Gap_(K+1)-s_(K+1)=1.6483


/* Gap Statistics of Hastie et al. */

data test;
     length ID 5;
     array _X{50} X1-X50;
  do ID=1 to 4000;
     beta=mod(ID, 5)+1;
  do _j=1 to 50;
     _X[_j]=rannor(6898776)+beta*3;
  end;
  output;
  keep ID X:;
  end;
run;


%macro GAPstat(dsn, ID_var, dsn_GAP, max_K, max_rep, samplemethod=1, seed=147852369);
%local dsn  B K dsid nobs ID dsn_out  features features2 num_features;

%let B=&max_rep;
%let dsid=%sysfunc(open(&dsn));
%let nobs=%sysfunc(attrn(&dsid, nobs));
%let dsid=%sysfunc(close(&dsid));
%put &nobs;

%let ID=&ID_var;
%let dsn_out=_clus_out;
%let ref_dist=_ref_dist;


proc contents data=&dsn out=allvars(keep=name varnum) noprint; run;
proc sort data=allvars; by varnum; run;
proc sql  noprint;
     select name into :features separated by ' '
  from   allvars
  where  name^="&ID"
  ;
     select count(distinct name) into :num_features 
  from   allvars
  where  name^="&ID"
  ;
quit;


%if &samplemethod eq 1 %then %do;
proc means data=&dsn noprint;
     var &features;
  output out=_lim(where=(_STAT_ in ('MIN', 'MAX')));
run;
%end;
%else %if &samplemethod eq 2 %then %do;
proc princomp data=&dsn   noint cov noprint
                     outstat=_V(where=(_TYPE_='USCORE'));
    var &features;
run;
data _V_score/view=_V_score;
       set _V;
    _TYPE_='PARMS';
    
run;
proc score data=&dsn  score=_V_score type=parms  out=_Z;
       var &features;
run;
proc means data=_Z  noprint;
       var &features;
    output   out=_lim(where=(_STAT_ in ('MAX', 'MIN')));
run;
%end;
%else %do;
         %put Only value=1 (uniformly sampling) and 2 (SVD sampling) allowed.;
         %goto exit;
%end;

data &ref_dist;
     array _stat{2, &num_features} _temporary_;
  array _F{&num_features}  &features;
  
     do until (eof);
     set _lim end=eof;
     if _STAT_='MIN' then _row=1;
     else _row=2;
     do _k=1 to &num_features; 
              if _row=1 then _stat[_row, _k]=_F[_k];
     else _stat[_row, _k]=_F[_k]-_stat[_row-1, _k];
     end;
 end;
  
     
  do _B=1 to &B;
     do &ID=1 to &nobs;
     do _k=1 to &num_features;
        _F[_k]=ranuni(&seed)*(_stat[2, _k]) + _stat[1, _k];
     end;    
     keep _B &ID &features;
     output;
  end;
 end;
run;

%if &samplemethod eq 2 %then %do;
proc transpose data=_V  out=_Vt;  run;
proc sql noprint;
       select  name into :features2 separated by ' ", "'
    from   allvars
    where  upcase(name)^=("&ID")
    ;
quit;
data _Vt_score/view=_Vt_Score;     
    set _Vt; 
    set allvars(where=(name in ("&features2")));
       retain  _TYPE_ "PARMS"; 
    _LABEL_=_NAME_;
    _NAME_=name;
    drop name;
run;
proc score data=&ref_dist   score=_Vt_score   type=parms
                out=&ref_dist.(keep=_B  &ID &features);
    var &features;
run;
%end;

%do K=1 %to &max_K;
    proc fastclus data=&dsn  maxclusters=&K  
              out=&dsn_out.(keep=&ID Distance CLUSTER) 
              noprint;
         var &features;
    run;

    proc fastclus data=&ref_dist maxclusters=&K  
              out=&dsn_out._&K.(keep=&ID _B Distance CLUSTER)  
              noprint;
         by _B;
         var &features;
    run;

 proc sort data=&dsn_out; by CLUSTER; run;
 proc sort data=&dsn_out._&K; by _B CLUSTER; run;

    proc means data=&dsn_out  sum noprint;          
      class CLUSTER;
         var Distance;
         output out=_dist_  sum(Distance)=Distortion;
    run;
    proc means data=&dsn_out._&K  sum noprint;
         by _B  CLUSTER;
         var Distance;
         output out=_dist_&K  sum(Distance)=Distortion;
    run;
    data _dist_; 
      set _dist_;
   _FREQ_=_FREQ_/&nobs;
 run;
 data _dist_&K;
      set _dist_&K;
   _FREQ_=_FREQ_/&nobs;
 run;
 proc means data=_dist_ noprint  sum;
      var Distortion;
   weight _FREQ_;
   output out=_dist_  sum(distortion)=distortion;
 run;
 proc means data=_dist_&K noprint  sum;
      by _B;
   var Distortion;
   weight _FREQ_;
   output out=_dist_&K  sum(distortion)=distortion;
 run;

    data _dist_&K(drop=mean _B:) _logmean_&K.(keep=mean);
         array _D{&B} _B1-_B&B;
         set _dist_&K end=eof;
         logD=log(Distortion)+log(&nobs);
         _D[_n_]=logD;
         output _dist_&k;
         if eof then do;     
            mean=mean(of _D[*]);
            output  _logmean_&K;  
         end;
    run;

    proc means data=_dist_&K  noprint;
         var logD;
         output out=_s_&K.(where=(_STAT_='STD'));
    run;

    data Gap_&K;
        set _dist_; set _logmean_&K ; set _s_&K(keep=logD);
  Wk=log(distortion)+log(&nobs);
        GAP=mean-Wk;
        s=logD*sqrt(1+1/&B);
        keep mean  Wk  GAP s;
    run;
%end;

data &dsn_GAP;
     set %do K=1 %to &max_K;
         Gap_&K
   %end;;
run;
%exit:
%mend;

%let dsn=test;
%let ID=ID;
%let dsn_GAP=GAP;
%let max_K=10;
%let max_rep=100;
%GAPStat(&dsn, &ID, &dsn_GAP, &max_K, &max_rep, samplemethod=2);

A reference paper can be found at:
R. Tibshirani, G. Walther, and T. Hastie. 2001.
Estimating the number of clusters in a dataset via the Gap statistic.
Journal of the Royal Statistics Society (Series B), pp. 411--423.

Or the book by R. Tibshirani, G. Walther, and T. Hastie:
The Elements of Statistical Learning

JMP Blog: Tools to Help Teach JMP

2010-02-17T14:30:00+00:00

In a recent live Webcast for faculty and students, I showed teaching modules developed by Amy Froelich (Iowa State University) and Bill Duckworth (Creighton University) that were programmed by Wayne Levin and Brian McFarlane (Predictum Inc.).

Here I show how to invoke these modules (which are encrypted JMP scripts) or other teaching scripts from a JMP menu you can create. Here is how your new menu might look:

Before you start:

If you don’t know how to add menus, please see these instructions.
The code for the modules or other teaching scripts need to reside on the computer from which you are running JMP.
To download the concept modules, you need a SAS profile. Log in only once per session to download all the modules, demos and files you want.
After you create and save the menu, you must close and reopen JMP for the scripts to work.

1. Download the scripts from http://www.jmp.com/academic/learning_modules.shtml to your hard drive.
2. Create a main Teaching menu and a sub-menu for each concept module. (After creation, the main menu will display on your menu bar, and when you expand the new main menu, you will see a new sub-menu that will execute your script.)
3. Name each sub-menu and associate it with the correct .jsl file: a. Right-click the sub-menu Untitled Menu and select Properties. A Menu Item Properties window appears.; b. Under Menu Item Type, select the Command radio button.; c. In the Caption box, type the name of your sub-menu.; d. Select the Run JSL in this file radio button.; e. Use the Browse button to locate and select the script file that you want this sub-menu to launch.; f. Click OK.
4. Add sample teaching scripts from the JMP Sample Script directory. For example, Mia finds the scripts for least squares and for comparing the normal and t distributions useful teaching tools.: a. Save the scripts to your hard drive. Typically, the scripts are stored in Program Files>SAS>JMP>8>Support Files> Sample Scripts.; b. Follow Step 3 to create sub-menus and select the script file to associate with each sub-menu.

NOTE: The blog of RTSL.eu: NOTE: The Missing Semicolon Just Arrived

2010-02-17T14:24:04+00:00

Systems Seminar Consultants' newsletter (named The Missing Semicolon) is always a good read, so I was pleased to get notification of the Winter 2010 issue last week. Featuring a mixture of topics, this issue seems to focus on writing good documentation (program documentation and system documentation). Please don't view this as a switch-off topic! Read the articles and you'll better understand the benefits that properly targeted and focused documentation offers.

However, I do strongly disagree with the author's rule of adding a comment to every line of code. Programming standards always give rise to a strong degree of discussion, but in my opinion slavishly putting comments onto every line of code doesn't add anything to the reader's knowledge of the code. Indeed, in the example code given, the vast majority of on-the-line comments are stating the obvious. Comments should describe what is not obvious in the code - that typically means describing what blocks of code are doing and/or why a particular approach was taken (and why other approaches were considered but discarded).

The issue also offers a review of The Little SAS Book (by Lora Delwich and Susan Slaughter whom I featured yesterday), and a nice tip regarding the INFILE statement's MISSOVER parameter.

I recommend you hop over to Systems Seminar Consultants' publications page and a) sign-up for a free subscription, and b) take some time to browse through the archive of issues.

SAS Programming: Tensor in SAS

2010-02-16T23:47:11+00:00

Tensor, a math concept for high order array, is a very useful tool in modern data mining applications. HOSVD, the counter part of SVD in higher order array, is at the heart of modern applications, such as face recognition and clustering, segmentation of multi-dimensional transaction data, etc.

【"Matrix Method in Data Mining and Pattern Recognition" by Lars Eldén】

But implementation of math operations designed for Tensor is not easy, not to mention doing that in SAS. Due to the design nature of SAS data set, everything has to be done in a 1-D or 2-D matrix fashion. Luckily, Tensor operations have their low dimension matrix counterpart, only that matrix subscriptions need to be calculated carefully.

I spent recent couple of days to implement some featured Tensor operations, notably Tensor unfolding (Tensor to Matrix) and HOSVD for my research and work. A prototype code for 3-mode Tensor unfolding and HOSVD SAS code is shown below. Data is from Lathauwer et al., "A MULTILINEAR SINGULAR VALUE DECOMPOSITION", SIAM J. MATRIX ANAL. APPL. vol.21, No.4



data A1;
     input X1-X3;
datalines;
 0.9073  0.8924  2.1488
 0.7158 -0.4898  0.3054
-0.3698  2.4288  2.3753
 1.7842  1.7753  4.2495
 1.6970 -1.5077  0.3207
 0.0151  4.0337  4.7146
 2.1236 -0.6631  1.8260
-0.0740  1.9103  2.1335
 1.4420 -1.7495 -0.2716
;
run;

filename Acat  catalog 'work.tensor.A2.CATAMS';

data _null_;
     file Acat;
     mode='1;';
     dimension='3,3,3;';
     put mode dimension;
run;

%let _dims=2 3 4;
%let _totalmode=3;
%let _mode0=1;
%let _modex=2;
%let _dimension2=6;
%let _dimension1=4;

data _null_;
      infile A2Cat;
      input;  
      _index=trim(scan(_infile_, 1, ';')); put _index=;
      _dimension=trim(scan(_infile_, 2, ';')); put _dimension=;       
      d2=count((_dimension), ',');  put d2=;
      call symput('_totalmode', d2+1);   
      call symput('_dims', translate(_dimension, ' ', ','));
      _dim0=scan(_dimension, &_mode0, ',');
      _dimx=scan(_dimension, &_modex, ',');
      call symput('_dim0', _dim0);
      call symput('_dimx', _dimx);
      call symput('_dim0x', compress(max(_dim0, _dimx)));
      dim2=1; dim1=1;
      do j=1 to d2+1;      
         if j=&_modex then dim1=dim1*scan(_dimension, j, ',');
         else dim2=dim2*scan(_dimension, j, ',');         
      end;
      put dim1= dim2= dim20=;
      call symput('_dimension1', compress(dim1));
      call symput('_dimension2', compress(dim2));   
run;
%put &_totalmode  &_dim0x;
%put &_dimension1  &_dimension2;
%put &_dims;

data  A&_modex;
      array _dim{&_totalmode} _temporary_ (&_dims);
      array _v{&_totalmode} _temporary_;
      array _datavec{&_dim0x} X1-X&_dim0x;
      array _X{&_dimension2, &_dimension1} _temporary_;
      set A&_mode0  end=eof;
      m1=mod(&_mode0+1, &_totalmode); if m1=0 then m1=&_totalmode;
      m2=mod(&_mode0+2, &_totalmode); if m2=0 then m2=&_totalmode;
      _v[m2]=mod(_n_, _dim[m2]); if _v[m2]=0 then _v[m2]=_dim[m2];
      _v[m1]=floor((_n_-_v[m2])/_dim[m2])+1;
      do j=1 to &_dim0;
         _v[&_mode0]=j;
         m1=mod(&_modex+1, &_totalmode); if m1=0 then m1=&_totalmode;
         m2=mod(&_modex+2, &_totalmode); if m2=0 then m2=&_totalmode;  
         _m=_v[&_modex];
         _n=(_v[m1]-1)*_dim[m2]+_v[m2];
         _X[_n, _m]=_datavec[j];
      end;
      if eof then do;
         do j=1 to &_dimension2;
            do k=1 to &_dimension1;
               _datavec[k]=_X[j, k];
            end;
            keep X1-X&_dim0x;
            output;
         end;
   end; 
run;

filename Acat  catalog "work.tensor.A&_modex..CATAMS";
data _null_;
     file Acat;
     mode="&_modex;";
     dimension="3,3,3;";
     put mode dimension;
run;

ods select none;
proc princomp data=A1  outstat=stat_1 noint  cov;
     var X1-X3;
run;
proc princomp data=A2  outstat=stat_2 noint  cov;
     var X1-X3;
run;
proc princomp data=A3  outstat=stat_3 noint  cov;
     var X1-X3;
run;
ods select all;

/***************************************
In order to obtain Tensor core S, we can 
do something like this:
S_(1)=t(U1)%*%A_(1)%*%(kronecker(U2, U3)
where S_(1) is 1-mode matrix of S and A_(1) 
is 1-mode matrix of A
****************************************/

%macro kroneck(mat_A, mat_B, mat_kronecker);
%local dsid row_B col_B  col_A;

%let dsid=%sysfunc(open(&mat_B));
%let row_B=%sysfunc(attrn(&dsid, nobs));
%let col_B=%sysfunc(attrn(&dsid, nvars));
%let dsid=%sysfunc(close(&dsid));
%let dsid=%sysfunc(open(&mat_A));
%let col_A=%sysfunc(attrn(&dsid, nvars));
%let dsid=%sysfunc(close(&dsid));

data _null_;
     col_K=&col_A*&col_B;
     call symput('col_K', compress(col_K));
run;
%put &col_A  &col_B  &col_K &row_B;

proc sql noprint;
     select name into :vars_A separated by ' '
  from   sashelp.vcolumn
  where  libname='WORK' and memname=%upcase("&mat_A")
  ;
     select name into :vars_B separated by ' '
  from   sashelp.vcolumn
  where  libname='WORK' and memname=%upcase("&mat_B")
  ;
quit;

data &mat_kronecker;
     array _K{&col_K} K1-K&col_K;
  array _A{&col_A} &vars_A;
  array _B{&col_B} &vars_B;
  array _Atmp{&col_A} _temporary_;
     array _Btmp{&col_B} _temporary_;

  do n1=1 to na;  
     set &mat_A  nobs=na point=n1;
  do j=1 to dim(_A);
           _Atmp[j]=_A[j];
  end;
     do n=1 to nb;
           set &mat_B nobs=nb point=n;
     do j=1 to dim(_B);
              _Btmp[j]=_B[j];
     end;
     do j=1 to &col_K;
        _ib=mod(j, &col_B); if _ib=0 then _ib=&col_B;
        _ia=floor((j-_ib)/&col_B)+1;
     *put _ia= _ib=  j= _A[_ia]= _B[_ib]=;
              _K[j]=_Atmp[_ia]*_Btmp[_ib];
     end;
     keep K1-K&col_K;
     output;
     end;
  end;
  stop;
run;

%mend kronecker;

options mprint  mlogic;
%kroneck(B, A, C);

data U1t(keep=X:);
     set stat_1;
     where _type_='USCORE';
run;
data U2t(keep=X:);
     set stat_2;
     where _type_='USCORE';
run;
data U3t(keep=X:);
     set stat_3;
     where _type_='USCORE';
run;

options nomprint  nomlogic;
%kroneck(U2t, U3t, U23t);

data A1score;
     set A1;
     retain _TYPE_ 'PARMS'; 
     _NAME_=compress('K'||_n_);
run;

proc score data=u1t  score=A1score out=u1A1(keep=K:) type=parms;
     var X1-X3;
run;
data u1A1;
     set u1A1;
     retain _TYPE_ 'PARMS'; 
     _NAME_=compress('X'||_n_);
run;

proc score data=u23t  score=u1A1  out=S1(keep=X:) type=parms;
     var K1-K9;
run;

Reference:
Matrix Methods in Data Mining and Pattern Recognition by Lars Eldén

StudySAS Blog: How to check if File is exist or not in SAS | StudySAS Blog [Digg]

2010-02-16T19:33:26+00:00

Guys… Let me try explaining how to check if the file exist in the library or directory using SAS. Here I am writing a macro to check if the file exist in the directory or not. Here is the way to check it…

StudySAS Blog: Renaming Variables Using the SASHELP Views| StudySAS Blog [Digg]

2010-02-16T18:22:04+00:00

1) The 1st step of the program determines the total number of variables inside the dataset with the help of SASHELP.Vtable, Data _null_ step used to do the same …. A macro variable NVAR (Number of Variables) will be created after this step which can be accessed in the following steps.

StudySAS Blog: Length of Numeric variables GT 8 in SAS| StudySAS BLOG [Digg]

2010-02-16T16:51:27+00:00

Q&A: numeric variables length more than 8? We all know that the default length of the numeric variables in SAS is 8 and if suppose I want to store a number lets say (12345678910, which has a length 11 to numeric variable) to variable total, what should I do?

StudySAS Blog: A Sampler of What's New in Base SAS 9.2 | StudySAS Blog [Digg]

2010-02-16T15:40:41+00:00

SAS 9.2, What's New in SAS 9.2

StudySAS Blog: Using ODS to Create Customised Output | StudySAS Blog [Digg]

2010-02-16T15:29:36+00:00

Using the SAS Output Delivery System (ODS), you can create, customise, and manage HTML output in any operating environment by submitting programming statements. After creating HTML files, you can view them using Internet Explorer, Netscape Navigator.....