Practical PowerShell for IT Security, Part IV:  Security Scripting Platform (SSP)

In the previous post in this series, I suggested that it may be possible to unify my separate scripts — one for event handling, the other for classification — into a single system. Dare I say it, a security platform based on pure PowerShell code?

After I worked out a few details, mostly having to do with migraine-inducing PowerShell events, I was able to declare victory and register my patent for SSP, the Security Scripting Platform ©.

United States of PowerShell

While I’m having an unforgettable PowerShell adventure, I realize that a few of you may not be able to recall my recent scripting handiwork.

Let’s review together.

In the first post, I introduced the amazing one line of PowerShell that watched for file events and triggered a PS-style script block — that is, a piece of scripting code than runs in its own memory space.

Register-WmiEvent -Query "SELECT * FROM __InstanceModificationEvent WITHIN 5 WHERE TargetInstance ISA 'CIM_DataFile' and TargetInstance.Path = '\\Users\\bob\\' and targetInstance.Drive = 'C:' and (targetInstance.Extension = 'doc' or targetInstance.Extension = 'txt)' and targetInstance.LastAccessed > '$($cur)' " -sourceIdentifier "Accessor" -Action $action

Register-WmiEvent -Query "SELECT * FROM __InstanceModificationEvent WITHIN 5 WHERE TargetInstance ISA 'CIM_DataFile' and TargetInstance.Path = '\\Users\\bob\\' and targetInstance.Drive = 'C:' and (targetInstance.Extension = 'doc' or targetInstance.Extension = 'txt)' and targetInstance.LastAccessed > '$($cur)' " -sourceIdentifier "Accessor" -Action $action

With a little more scripting sauce, I then cooked up what I called a File Access Analytics (FAA) app. Effectively, it tallies up the access event and displays some basic stats, and also detects bursts of access activity, which could indicate hacking behavior.

It’s a simplified version of User Behavior Analytics (UBA) technology that we’re keen on here at Varonis.

So far, so good.

Then in the third post, I showed how relatively easy it is with PowerShell to scan and classify files in a folder. Since this is a disk-intensive activity, it makes incredible sense to use PowerShell’s multi-tasking capability, known as Runspaces to speed up the classification work.

In the real-world of file event handling and data classification, say Varonis’s Data Classification Framework, a more optimized approach to categorizing file content is to feed file events into the classification engine.

Why?

Because then you don’t have to reclassify file content from scratch: you only look at the files that have changed. My classifier script would therefor benefit greatly by knowing something about file modification events.

That’s the approach I took with SSP.

Varonis’s own agents, which catch Linux or Windows file events, are finely tuned low-level code. For this kind of work, you want the code to be lean, mean, and completely focused on collecting events and to quickly pass this info to other apps that can do higher-level processing.

So I took my original event handling script, streamlined it and removed all the code to display statistics. I then reworked the classifier to scan for specific files that have been modified.

Basically, it’s a classic combination of a back-end engine coupled with a front-end interface.

The question is how to connect the two scripts: how do I tell the classifier that there’s been a file event?

Messages and Events

After spending a few long afternoons scanning the dev forums, I eventually stumbled upon PowerShell’s Register-EngineEvent.

What is this PowerShell cmdlet?

In my mind, it’s a way to pass messages using a named event that you can share between scripts. It works a little bit differently from traditional system messaging and queues since the received message asynchronously triggers a PowerShell script block. This’ll become clearer below.

In any case, register-EngineEventregister-EngineEvent has two faces. With the -forward-forward parameter, it acts as a publisher.  And without the -forward ,-forward ,parameter it takes on the role of a receiver.

Got that?

I used the event name Delta — technically the SourceIdentifer — to coordinate between my event handler script, which pushes out event messages, and my classifier script, which receives these message.

In the first of two scripts snippets below, I show how I register the public Delta event name with -Register-EngineEvent -forward-Register-EngineEvent -forward, and then wait for internal file access events. When one comes in, I then send the internal file event message — in PowerShell-speak, it’s forwarded — to the corresponding Register-EngineEventRegister-EngineEventin the classifier script in the second snippet.

Register-EngineEvent -SourceIdentifier Delta -Forward
While ($true) {
   $args=Wait-Event -SourceIdentifier Access  # wait on internal file event
    Remove-Event -SourceIdentifier Access
    if ($args.MessageData -eq "Access") {  
       #do some plain access processing 
       New-Event -SourceIdentifier Delta -EventArguments $args.SourceArgs -MessageData $args.MessageData  #send event to classifier via forwarding
     }
    elseif ($args.MessageData -eq "Burst") {
       #do some burst processing
       New-Event -SourceIdentifier Delta -EventArguments $args.SourceArgs  -MessageData $args.MessageData #send event to classifier via forwarding
     }
}

Register-EngineEvent -SourceIdentifier Delta -Forward
While ($true) {
   $args=Wait-Event -SourceIdentifier Access  # wait on internal file event
    Remove-Event -SourceIdentifier Access
    if ($args.MessageData -eq "Access") {  
       #do some plain access processing 
       New-Event -SourceIdentifier Delta -EventArguments $args.SourceArgs -MessageData $args.MessageData  #send event to classifier via forwarding
     }
    elseif ($args.MessageData -eq "Burst") {
       #do some burst processing
       New-Event -SourceIdentifier Delta -EventArguments $args.SourceArgs  -MessageData $args.MessageData #send event to classifier via forwarding
     }
}

On the receiving side, I leave out the -forward -forward parameter and instead pass in a PowerShell script bock, which asynchronously handles the event. You can see this below.

Register-EngineEvent -SourceIdentifier Delta -Action {
    
      Remove-Event -SourceIdentifier Delta
      if($event.MessageData -eq "Access") {
        $filename = $args[0] #got file!
         Lock-Object $deltafile.SyncRoot{ $deltafile[$filename]=1} #lock&load            
      }
      elseif ($event.Messagedata -eq "Burst") {
        #do something     
      }

}

Register-EngineEvent -SourceIdentifier Delta -Action {
    
      Remove-Event -SourceIdentifier Delta
      if($event.MessageData -eq "Access") {
        $filename = $args[0] #got file!
         Lock-Object $deltafile.SyncRoot{ $deltafile[$filename]=1} #lock&load            
      }
      elseif ($event.Messagedata -eq "Burst") {
        #do something     
      }

}

Confused? And have I mentioned recently that file event handling is not easy, and that my toy scripts won’t stand up to business-level processing?

This gets messy because the New-Event New-Event and Wait-EventWait-Eventcmdlets for internal event messaging are different from the external event messaging provided by Register-EngineEvent. Register-EngineEvent.

More Messiness

The full classification script is presented below. I’ll talk a little more about it in the next and final post in this series. In the meantime, gaze upon it in all its event-handling and multi-tasking glory.

Import-Module -Name .\pslock.psm1 -Verbose
function updatecnts {
Param ( 
        [parameter(position=1)]  
        $match, 
        [parameter(position=2)]
        $obj
        )

for($j=0; $j -lt $match.Count;$j=$j+2) {    
        switch -wildcard ($match[$j]) {
          'Top*'  { $obj| Add-Member -Force -type NoteProperty -Name Secret   -Value $match[$j+1] }
          'Sens*' { $obj|  Add-Member -Force -type NoteProperty -Name Sensitive -Value $match[$j+1] }
          'Numb*' { $obj|  Add-Member -Force -type NoteProperty -Name Numbers  -Value $match[$j+1] }           
         }
         
      }

  return $obj
}
  
$scan = {
$name=$args[0]
function scan {
   Param (
      [parameter(position=1)]
      [string] $Name
   )
      $classify =@{"Top Secret"=[regex]'[tT]op [sS]ecret'; "Sensitive"=[regex]'([Cc]onfidential)|([sS]nowflake)'; "Numbers"=[regex]'[0-9]{3}-[0-9]{2}-[0-9]{3}' }
     
      $data = Get-Content $Name
      
      $cnts= @()
      
      if($data.Length -eq 0) { return $cnts} 
      
      foreach ($key in $classify.Keys) {
       
        $m=$classify[$key].matches($data)           
           
        if($m.Count -gt 0) {
           $cnts+= @($key,$m.Count)  
        }
      }   
 $cnts   
}
scan $name
}

 
$outarray = @() #where I keep classification stats
$deltafile = [hashtable]::Synchronized(@{})  #hold file events for master loop 

$list=Get-WmiObject -Query "SELECT * From CIM_DataFile where Path = '\\Users\\bob\\' and Drive = 'C:' and (Extension = 'txt' or Extension = 'doc' or Extension = 'rtf')"  


#long list --let's multithread

#runspace
$RunspacePool = [RunspaceFactory]::CreateRunspacePool(1,5)
$RunspacePool.Open()
$Tasks = @()


foreach ($item in $list) {
  
  $Task = [powershell]::Create().AddScript($scan).AddArgument($item.Name)
  $Task.RunspacePool = $RunspacePool
  
  $status= $Task.BeginInvoke()
  $Tasks += @($status,$Task,$item.Name)
}


Register-EngineEvent -SourceIdentifier Delta -Action {
    
      Remove-Event -SourceIdentifier Delta
      if($event.MessageData -eq "Access") {
        $filename = $args[0] #got file
         Lock-Object $deltafile.SyncRoot{ $deltafile[$filename]=1} #lock& load
      }
      elseif ($event.Messagedata -eq "Burst") {
        #do something
      }
}

while ($Tasks.isCompleted -contains $false){
  
}

#check results of tasks
for ($i=0; $i -lt $Tasks.Count; $i=$i+3){
   $match=$Tasks[$i+1].EndInvoke($Tasks[$i])
   
  
   if ($match.Count -gt 0) {  # update clasafication array 
      $obj = New-Object System.Object
      $obj | Add-Member -type NoteProperty -Name File   -Value $Tasks[$i+2]
      #defaults
      $obj| Add-Member -type NoteProperty -Name Secret -Value 0
      $obj| Add-Member -type NoteProperty -Name Sensitive -Value 0
      $obj| Add-Member -type NoteProperty -Name Numbers -Value 0

      $obj=updatecnts $match $obj
      $outarray += $obj
   } 
   $Tasks[$i+1].Dispose()
   
}

$outarray | Out-GridView -Title "Content Classification" #display

#run event handler as a separate job
Start-Job -Name EventHandler -ScriptBlock({C:\Users\bob\Documents\evhandler.ps1})  #run event handler in background


while ($true) { #the master executive loop
   
   
      Start-Sleep -seconds 10
      Lock-Object $deltafile.SyncRoot { #lock and iterate through synchronized list
        foreach ($key in $deltafile.Keys) {  
    
        $filename=$key
       
        if($deltafile[$key] -eq 0) { continue} #nothing new

        $deltafile[$key]=0
        $match = & $scan $filename  #run scriptblock
                                        #incremental part
       
        $found=$false
        $class=$false
        if($match.Count -gt 0) 
            {$class =$true} #found sensitive data
        if($outarray.File -contains $filename) 
                {$found = $true} #already in the array  
        if (!$found -and !$class){continue}
 
        #let's add/update
        if (!$found) {

            $obj = New-Object System.Object
            $obj | Add-Member -type NoteProperty -Name File   -Value $Tasks[$i+2]
            #defaults
            $obj| Add-Member -type NoteProperty -Name Secret -Value 0
            $obj| Add-Member -type NoteProperty -Name Sensitive -Value 0
            $obj| Add-Member -type NoteProperty -Name Numbers -Value 0

            $obj=updatecnts $match $obj

        }
        else {
            $outarray|? {$_.File -eq $filename} | % { updatecnts $match $_} 
        }
        $outarray | Out-GridView -Title "Content Classification ( $(get-date -format M/d/yy:HH:MM) )"   
        
       } #foreach

    } #lock
}#while

Write-Host "Done!"

Import-Module -Name .\pslock.psm1 -Verbose
function updatecnts {
Param ( 
        [parameter(position=1)]  
        $match, 
        [parameter(position=2)]
        $obj
        )

for($j=0; $j -lt $match.Count;$j=$j+2) {    
        switch -wildcard ($match[$j]) {
          'Top*'  { $obj| Add-Member -Force -type NoteProperty -Name Secret   -Value $match[$j+1] }
          'Sens*' { $obj|  Add-Member -Force -type NoteProperty -Name Sensitive -Value $match[$j+1] }
          'Numb*' { $obj|  Add-Member -Force -type NoteProperty -Name Numbers  -Value $match[$j+1] }           
         }
         
      }

  return $obj
}
  
$scan = {
$name=$args[0]
function scan {
   Param (
      [parameter(position=1)]
      [string] $Name
   )
      $classify =@{"Top Secret"=[regex]'[tT]op [sS]ecret'; "Sensitive"=[regex]'([Cc]onfidential)|([sS]nowflake)'; "Numbers"=[regex]'[0-9]{3}-[0-9]{2}-[0-9]{3}' }
     
      $data = Get-Content $Name
      
      $cnts= @()
      
      if($data.Length -eq 0) { return $cnts} 
      
      foreach ($key in $classify.Keys) {
       
        $m=$classify[$key].matches($data)           
           
        if($m.Count -gt 0) {
           $cnts+= @($key,$m.Count)  
        }
      }   
 $cnts   
}
scan $name
}

 
$outarray = @() #where I keep classification stats
$deltafile = [hashtable]::Synchronized(@{})  #hold file events for master loop 

$list=Get-WmiObject -Query "SELECT * From CIM_DataFile where Path = '\\Users\\bob\\' and Drive = 'C:' and (Extension = 'txt' or Extension = 'doc' or Extension = 'rtf')"  


#long list --let's multithread

#runspace
$RunspacePool = [RunspaceFactory]::CreateRunspacePool(1,5)
$RunspacePool.Open()
$Tasks = @()


foreach ($item in $list) {
  
  $Task = [powershell]::Create().AddScript($scan).AddArgument($item.Name)
  $Task.RunspacePool = $RunspacePool
  
  $status= $Task.BeginInvoke()
  $Tasks += @($status,$Task,$item.Name)
}


Register-EngineEvent -SourceIdentifier Delta -Action {
    
      Remove-Event -SourceIdentifier Delta
      if($event.MessageData -eq "Access") {
        $filename = $args[0] #got file
         Lock-Object $deltafile.SyncRoot{ $deltafile[$filename]=1} #lock& load
      }
      elseif ($event.Messagedata -eq "Burst") {
        #do something
      }
}

while ($Tasks.isCompleted -contains $false){
  
}

#check results of tasks
for ($i=0; $i -lt $Tasks.Count; $i=$i+3){
   $match=$Tasks[$i+1].EndInvoke($Tasks[$i])
   
  
   if ($match.Count -gt 0) {  # update clasafication array 
      $obj = New-Object System.Object
      $obj | Add-Member -type NoteProperty -Name File   -Value $Tasks[$i+2]
      #defaults
      $obj| Add-Member -type NoteProperty -Name Secret -Value 0
      $obj| Add-Member -type NoteProperty -Name Sensitive -Value 0
      $obj| Add-Member -type NoteProperty -Name Numbers -Value 0

      $obj=updatecnts $match $obj
      $outarray += $obj
   } 
   $Tasks[$i+1].Dispose()
   
}

$outarray | Out-GridView -Title "Content Classification" #display

#run event handler as a separate job
Start-Job -Name EventHandler -ScriptBlock({C:\Users\bob\Documents\evhandler.ps1})  #run event handler in background


while ($true) { #the master executive loop
   
   
      Start-Sleep -seconds 10
      Lock-Object $deltafile.SyncRoot { #lock and iterate through synchronized list
        foreach ($key in $deltafile.Keys) {  
    
        $filename=$key
       
        if($deltafile[$key] -eq 0) { continue} #nothing new

        $deltafile[$key]=0
        $match = & $scan $filename  #run scriptblock
                                        #incremental part
       
        $found=$false
        $class=$false
        if($match.Count -gt 0) 
            {$class =$true} #found sensitive data
        if($outarray.File -contains $filename) 
                {$found = $true} #already in the array  
        if (!$found -and !$class){continue}
 
        #let's add/update
        if (!$found) {

            $obj = New-Object System.Object
            $obj | Add-Member -type NoteProperty -Name File   -Value $Tasks[$i+2]
            #defaults
            $obj| Add-Member -type NoteProperty -Name Secret -Value 0
            $obj| Add-Member -type NoteProperty -Name Sensitive -Value 0
            $obj| Add-Member -type NoteProperty -Name Numbers -Value 0

            $obj=updatecnts $match $obj

        }
        else {
            $outarray|? {$_.File -eq $filename} | % { updatecnts $match $_} 
        }
        $outarray | Out-GridView -Title "Content Classification ( $(get-date -format M/d/yy:HH:MM) )"   
        
       } #foreach

    } #lock
}#while

Write-Host "Done!"

In short, the classifier does an initial sweep of the files in a folder, stores the classification results in $outarray$outarray, and then when a file modification event shows up, it updates and displays $outarray$outarray with new classification data. In other words, incremental scanning.

There’s a small side issue of having to deal with updates to $outarray$outarray that can happen at any time while in another part of the classification script I’m actually looking to see what’s changed in this hashtable variable.

It’s a classic race condition. And the way I chose to handle it is to use PowerShell’s synchronized variables.

I’ll talk more about this mystifying PowerShell feature in the next post, and conclude with some words of advice on rolling-your-own solutions.