Akamai Datastream Logs to ELK Stack via API

 If you are using the security features of Akamai, using web security analytics(WSA) and SIEM log delivery features of Akamai  may be enough for dealing with daily security events. For CDN features there is  traffic and url traffic reports up to 90 days. You can also pull the logs via datastream api  and parse them with a Logstash json filter, output it to a Elasticsearch instance. Finally query, and present the parsed logs in a Kibana  dashboard. 

So to start you should follow the below steps. (Follow the links in the steps)

• Enable Datastream with the desired fields from Luna Control Center
• Create api privileged user
• Test your user with the credential you have created in the above step with an api client like Postman
• Follow the instructions to become familiar with the datastream api 
• Now use this python code to fetch logs from a datastream and write the logs to a file that will be parsed via Logstash in the below steps. 
• Schedule/Cronjob the python script to run in every 5 minutes.
• Follow the instructions here to install ELK stack but do not configure logstash conf file yet. 
• Use this logstash config file to parse the datastream pulled json files. 
• Start all ELK services logstash, elasticsearch, kibana
• Create Kibana index patttern 
• Verify that logs are parsed properly and can be seen on Kibana Discovery application
• Using the Lens app in Kibana create your visualizations to build up a dashboard. 

Creating proxy pac files automatically from Squid Conf file.

Pac files are automatic proxy configuration files written in js format that browsers can interpret and route clients to the proxy according to the domain name they visit. This pac file should be published through a web server or can be manually downloaded and the browser should be configured to the downloaded pac file. Squid is an open source proxy server which can be configured to be used by libraries for out off campus access to the academic database subscriptions . When someone add/delete/update an entry in the squid conf file, pac files should also be updated properly. This is a tedious task for large numbers of urls. Python script which automates this task with per minute cron entry

The script pushes syslogs to the syslog of the system on every execution. If any changes made to the squid file, the pac file is updated by checking the hash of the old and new states of the file. Also the squid proceess is restarted if any change has been made.

import hashlib
import os
import syslog
import datetime
import shutil
import subprocess
squid_config_file_path="/etc/squid/kutuphane_veritabanlari.squid"
squid_config_file = open(squid_config_file_path)
readFile = squid_config_file.read()
pac_file_root="/var/www/html"
hash_file="%s/hash_file"%pac_file_root
old_pac_file="%s/proxy.pac"%pac_file_root
date_=datetime.datetime.now() 
backup_file="%s/pac_backup_file_%s.pac"%(pac_file_root,date_)
new_pac_filee="%s/new_pac_file.pac"%pac_file_root
sha1Hash = hashlib.sha1(readFile)
sha1Hashed = sha1Hash.hexdigest()
if os.path.exists(hash_file):
   hash_filee = open(hash_file,'r')
   readHash = hash_filee.readline()
   hash_filee.close()
   if not readHash==sha1Hashed:
 syslog.syslog(syslog.LOG_WARNING,"Hashes are different  kutuphane_veritabanlari.squid config file has been modified the old hash :%s new hash:%s "%(readHash,sha1Hashed))
 shutil.copyfile(old_pac_file,backup_file) 
 new_pac_file = open(new_pac_filee,"w+")
    new_pac_file.write("function FindProxyForURL(url, host) {\n")
 new_pac_file.write("var proxyserver = 'proxy2.iyte.edu.tr:8080';\n")    
 new_pac_file.write("var proxylist = new Array(\n")
    with open(squid_config_file_path) as openfileobject:
           for line in openfileobject:
          if (line[0].isalpha() or line.startswith(".")):
   new_pac_file.write('"%s",\n' % (line.rstrip()))
        new_pac_file.write('"dummy.com"\n);\n')
    new_pac_file.write("for(var i=0; i<proxylist.length; i++) {\n")
    new_pac_file.write("\tvar value = proxylist[i];\n")
    new_pac_file.write("\t\tif (dnsDomainIs(host, value) ) {\n")
    new_pac_file.write("\t\treturn 'PROXY '+proxyserver;\n")
    new_pac_file.write("\t}\n")
    new_pac_file.write("}\n")
    new_pac_file.write("return 'DIRECT';\n")
    new_pac_file.write("}")
 new_pac_file.close()
 os.remove(hash_file) 
 hash_file_create=open(hash_file,'w')
     hash_file_create.write(sha1Hashed)
     hash_file_create.close() 
 shutil.copyfile(new_pac_filee,old_pac_file)
 subprocess.call("%s %s %s" % ('service', 'squid', 'reload'),shell=True)
   else:
 syslog.syslog(syslog.LOG_WARNING,"Hashes are same kutuphane_veritabanlari.squid config file hasn't been changed.")
else:
    hash_file_create=open(hash_file,'w')
    hash_file_create.write(sha1Hashed)
    hash_file_create.close()
squid_config_file.close()