I focus on discussion of association based upon which, if you introduce temporal order of the transaction, you can easily extend /imagine into sequence.
The SASIMSH system used for this post is the same as the one used for my post dated 12/14/2014 "SAS High Performance Analytics and In-Memory Statistics for Hadoop: Two Genuine in-Memory Math Blades Working Together". Here are some info on the data set used.
The data set is simulated transaction data set consisting of 12 monthly transaction, 25 million transaction entries each, totaling 300 millions. The total size of the data set is ~125 GB. Below is monthly distribution.
T_weekday is how many transactions happen Sunday, Monday, Tuesday... Saturday. T_week counts how many transactions happen on week 1... week24....week52 on the year. These segment variables are created in case you want to break down your analysis.
Below is main body of the ARM modeling code
1. The two "Proc LASR" sections create LASR in-memory analytics process and load the analytics data set into it. The creation process took ~10 seconds and the loading process took ~15 seconds (see picture)
2. The Frequency statement simply profiles the variables the distributions of which I reported above.
3. The ARM statement is where the core activities happen
- Item= is where you list the variable of product category. You have full control product hierarchy.
- Tran= is where you specify granular level of transaction data. There are ~9 million unique accounts for this exercise. If you choose to use a level that has, say, 260 unique level values (with proper corresponding product levels) you can easily turn the ARM facility into BI reporting tool, closer to IMSTAT's GROUPBY statement does.
- You can use MAXITEMs= (and/or MINITEMS) to customize item counts for compilation
- Freq = is simply order count of the item. While Freq = is more 'physical, accounting book weight' (therefore less analytical, by definition), Weight= weighting is more analytical /intriguing. I used list price here, essentially compiling support in terms of individual price importance, assuming away any differential price-item elasticity and a lot more. You can easily have a separate model to study this weight input alone, which is beyond the scope of this post.
- The two aggregation options allow you to decide how item aggregation and ID aggregation should happen; if weight = is left blank, both aggregations ignore the aggregation= values you plug in and aggregate by default value of SUM, which is really to ADD UP. Ideally, one aggregation should use one weight variable. For now, if you specify weight=, the weigh variable is used for both. If you are really so 'weight' sensitive, you can run the aggregation one at a time, which does not much more time and resources.
- The ITEMSTBL option asks output of a temporary table to be created in-memory amid the flow for further actions during the in-memory process, the table system-reserved keyword .&_tempARMItems_ refers to in the next step. This is different from what SAVE option generates. SAVE typically outputs table to Hadoop directory "when you are done".
- The list of options commented out in GREEN show that you can customize support output; you don't have to follow the same configurations when the ARM model was being fit above when generating rules or association scores.
- The _T_ table is the temporary table created. You can use PROMOTE statement to make it permanent
- _SetSize_ simply tells number of products in the combinations.
- _Score_ is the result of your (double) aggregations. Since you can select one of 4 aggregation
- options (SUM, MEAN, MIN, MAX) for either aggregation (ITEMAGG and AGG), you need to interpret the score according to your options.
5. This whole, while sounding cliche content wise, takes only ~8 minutes to finish over 300 million rows.
The gap between CPU time and real time is pretty large, but I care less since the overall is only 8 minutes.
