Install Apache Solr as Windows Service using NSSM

Today I wanted to install Apache Solr index in Windows 7, Windows Server 2008R2 and Windows Server 2012 machines.
So I achieved that , using the following steps:

  1. Download Apache Solr
  2. Download NSSM – the Non-Sucking Service Manager
  3. Create a .bat init file for Solr
  4. Create Windows service using NSSM
  5. Test Solr service

In detail:

  1. Download Apache Solr

    Download Apache Solr .zip file from Apache Solr website. I am currently using version 4.6.0.

  2. Download NSSM – the Non-Sucking Service Manager

    After a lot of searching for a stable solution in creating Windows services I stumbled upon NSSM – the Non-Sucking Service Manager. Download it and place the nssm.exe executable in a directory that can be seen by $PATH.

  3. Create a .bat init file for Solr

    1. Unzip Apache Solr zip file to
      C:\Program Files\apache-solr-4.6.0\
    2. Create C:\Program Files\apache-solr-4.6.0\example\solrstart.bat file containing the following:

      					@echo off
      					cd "C:\Program Files\apache-solr-4.6.0\example\"
         					java -Xms64M -Xmx256M  -Djava.util.logging.config.file=etc/logging.properties -jar start.jar
      					
  4. Create Windows service using NSSM

    1. Open a command line prompt (WinKey + R and type “cmd”) and type the following

      			nssm install Solr4.6.0 "C:\Program Files\apache-solr-4.6.0\example\solrstart.bat"
      			

      If the procedure succeeds you should see a message like this:Service “Solr4.6.0” installed successfully!

    2. If you want to uninstall the service open again a command line prompt and type the following:

      			nssm remove Solr4.6.0 confirm
      			

      If the procedure succeeds you should see a message like this:Service “Solr4.6.0” removed successfully!

  5. Test Solr service

    1. Open a command line prompt (WinKey + R and type “cmd”) and type the following

      			net start Solr4.6.0
      			
    2. If the service was started successfully, open a web browser and access admin page: http://localhost:8983

That is all about it folks! May the Solr be with you!

Posted in Apache, Solr, Windows | Tagged , , , , , , , , | 2 Comments

Remove special and control characters from string using Python

Today I had a database table which contained special and control characters (like DLE, DC1, ACK, DC3 etc.) in its records. After some searching I found a way to substitute those characters with their respective unicode escape value. And here Python comes to the rescue again!

#!/usr/bin/python
# -*- coding: utf-8 -*-

# DLE, DC1, ACK, DC3 characters
# a string with special and control characters
bob = """`Αδεια διεξαγωγής αγώνων κυνηγετικών ικανοτήτων σκύλων δεικτών."""

# bob string encoded in escaped unicode string
escapedbob = "\u0010`\u0391\u03b4\u03b5\u03b9\u03b1 \u03b4\u03b9\u03b5\u03be\u03b1\u03b3\u03c9\u03b3\u03ae\u03c2 \u03b1\u03b3\u03ce\u03bd\u03c9\u03bd \u03ba\u03c5\u03bd\u03b7\u03b3\u03b5\u03c4\u03b9\u03ba\u03ce\u03bd \u03b9\u03ba\u03b1\u03bd\u03bf\u03c4\u03ae\u03c4\u03c9\u03bd \u03c3\u03ba\u03cd\u03bb\u03c9\u03bd \u03b4\u03b5\u03b9\u03ba\u03c4\u03ce\u03bd.\u0010\u0011\u0006\u0013\u0010" 
def main():
    
    # encode special characters to unicode-escape
    print bob.decode('utf-8').encode('unicode-escape')

    # decode unicode-escape to simple characters
    print escapedbob.decode('unicode-escape')

    # exit
    exit(0)

if __name__ == '__main__':
    main()


# used tutorial
# http://stackoverflow.com/questions/10268518/python-string-to-unicode

Good luck tampering and hacking with your database strings !

Posted in Linux, Python | Tagged , , , , , , , | Leave a comment

Read SMS directly from Sqlite database in Android

Today I wanted to extract the SMS content out of the android SMS application SQLite database using ADB.

NOTE: The whole procedure was tested on a rooted phone running Cyanogenmod Android. It may not work on unrooted phones

So I achieved that , using the following steps:

  1. Find SMS database file location
  2. Pull SQLite database file with ADB
  3. Identify sms tables
  4. Identify the thread _ID of the SMS you want to read
  5. Find all sms messages for a specific thread id

In detail:

  1. Find SMS database file location

    After some searching I found that the location of sms database file is in

    /data/data/com.android.providers.telephony/databases/mmssms.db
  2. Pull SQLite database file with ADB

    This step is optional. Open a shell and type:

    adb pull /data/data/com.android.providers.telephony/databases/mmssms.db mmssms.db
    

    Now the whole sms database is copied locally in your working directory!

  3. Identify sms tables

    Open the SQLite using command line or your favourite SQLite editor. The tables in which the android telephony application stores SMS messages are threads and sms.
    Table Threads stores a record as a header for every SMS thread which is started:

    CREATE TABLE threads (
       _id INTEGER PRIMARY KEY AUTOINCREMENT,
       date INTEGER DEFAULT 0,message_count INTEGER DEFAULT 0,
       recipient_ids TEXT,snippet TEXT,
       snippet_cs INTEGER DEFAULT 0,
       read INTEGER DEFAULT 1,
       type INTEGER DEFAULT 0,
       error INTEGER DEFAULT 0,
       has_attachment INTEGER DEFAULT 0
    )
    

    Table SMS stores all outgoing and incoming messages for every thread:

    CREATE TABLE sms (_id INTEGER PRIMARY KEY,
       thread_id INTEGER,
       address TEXT,
       person INTEGER,
       date INTEGER,
       date_sent INTEGER DEFAULT 0,
       protocol INTEGER,
       read INTEGER DEFAULT 0,
       status INTEGER DEFAULT -1,
       type INTEGER,
       reply_path_present INTEGER,
       subject TEXT,
       body TEXT,
       service_center TEXT,
       locked INTEGER DEFAULT 0,
       error_code INTEGER DEFAULT 0,
       seen INTEGER DEFAULT 0
    )
    
  4. Identify the thread _ID of the SMS you want to read

    Run a SQL query and read snippet column. This column stores the last sms in your thread and is the visible text of every SMS list item when you open the SMS application.

    SELECT _id, snippet
    FROM threads;
    
  5. Find all sms messages for a specific thread id

    In my case the thread _id had the value 310. Knowing this id we run another query on sms table in order to get SMS messages ordered chronologically by date:

    SELECT 	datetime(date/1000, 'unixepoch','localtime')  ,datetime(date_sent/1000, 'unixepoch','localtime') ,person,body
    FROM sms
    WHERE thread_id = 310
    ORDER BY date
    

    The results are going to be similar to the following:

    +-----------------------------------------------------------------------------------------------------------------+
    | date                | date_sent             | person    |  body						  |
    |------------------------------------------------------------------------------------------------------------------
    | 2013-10-20 13:48:18 | 2013-10-20 13:48:16   |	  54	  | Hello Christos! How are you?			  |
    | 2013-10-20 16:34:03 | 1970-01-01 02:00:00   | 	  | Fine, thanks ! I configure the left MFD of a F-16 jet |
    | 2013-10-20 16:40:02 | 2013-10-20 16:40:01   |	  54	  | Awesome! I am throwing a party tomorrow at 21:45!	  |
    | 2013-10-20 17:15:15 | 1970-01-01 02:00:00   |		  | Thanks! I will be there!				  |
    +-----------------------------------------------------------------------------------------------------------------+
    

    Note that person with id 54 is my friend Kitsos and person = NULL is me!

Good luck tampering and hacking with your Android device !

Posted in Android, Databases, Linux, SQLite | Tagged , , , , , , , , , , , , | 28 Comments

Linux get hard disk statistics using Python

Today I wanted to collect disk usage information and send them to a remote server script. I used df command to collect disk usage information and Python to send them to a remote server. Note that the remote server script uses basic authentication. So, here is the script which does the dirty job:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

# used the following links:
# http://docs.python.org/2/library/re.html
# http://docs.python.org/2/library/httplib.html
# http://mozgovipc.blogspot.gr/2012/06/python-http-basic-authentication-with.html
# http://stackoverflow.com/questions/4760215/running-shell-command-from-python-and-capturing-the-output

import sys
import subprocess
import re
import httplib
import urllib
import base64
import logging

authUser = "myuser"
authPass = "mypadd"
logFile = "./sendDiskData.log"

def main():

    # initialise logging
    logging.basicConfig(level=logging.DEBUG,
                    format='%(asctime)s %(levelname)s %(message)s',
                    filename=logFile,
                    filemode='w')

    # execute df -H
    p = subprocess.Popen(['df', '-H'], stdout=subprocess.PIPE, 
                                       stderr=subprocess.PIPE)
    # get stdout and stderr
    out, err = p.communicate()

    # find /dev/vda1
    mato = re.findall(r'/dev/[sv]da.*',out,re.MULTILINE)

    # execute hostname
    p = subprocess.Popen(['hostname'], stdout=subprocess.PIPE, 
                                       stderr=subprocess.PIPE)
    # get stdout and stderr
    out, err = p.communicate()

    # get Hostname
    hostName = out


    # check if object is None (null)
    if mato:
        diskData = re.sub(r'^(.dev..da[^ \t]*)[ \t]*([0-9]*[.]*[0-9]*[MKG])[ \t]*([0-9]*[.]*[0-9]*[MKG])[ \t]*([0-9]*[.]*[0-9]*[MKG])[ \t]*([0-9]*[.]*[0-9]*[%]).*',r'\1,\2,\3,\4,\5',mato[0])
    
    # create basic authentication header 
    auth = base64.encodestring('%s:%s' % (authUser, authPass)).replace('\n', '')
    
    # create request params
    params = urllib.urlencode({'hostname': hostName, 'diskdata': diskData})

    # add Basic Authentication Header
    headers = {"Authorization": "Basic %s" % auth}
    
    # create url string
    diavUrl = "/clusterdisks.php?%s" % params

    try:
        conn = httplib.HTTPConnection("manios.org")
    
        # perform request
        conn.request("GET", diavUrl , None, headers)

        response = conn.getresponse()

        logging.info("Disk data sent. Code:%s , Reason:%s",response.status, response.reason)
    except HTTPException :
        #logging.error("Failed to send disk data of %s: with error : %s",hostName,e1)
        logging.error("Failed to send disk data of %s",hostName)
        
        sys.exit(1)

    # Exit with success
    sys.exit(0)

if __name__ == '__main__':
    main()
Posted in Linux, Python, Ubuntu | Tagged , , , , , , , | Leave a comment

Linux mount Windows shared directory via cifs and samba

Updates:

  • 2013-08-08: Added sec=ntlm
  • 2014-09-04: Added domain=mydomain parameter in order to connect to Active Directory domain

Today I wanted to mount a Windows shared directory on my Ubuntu Linux machine. Note that the Windows machine is connected to an Active directory domain. After some searching in the web and trying various solutions, I stumbled upon an answer in Ubuntu forums. I finally managed to mount the Windows directory using mount command and having installed Linux cifs utilities.
Important note: Verify that your Windows shared directory has read access or write access for the user you are connecting with.

So in order to get your job done follow these steps:

  1. Determine your Windows share directory information
    In my case they are:

    • Active Directory machine name: manios.mycompany.local
    • IP: 192.168.1.75
    • Shared Directory: books
    • Active Directory username: manios
    • Active Directory password: maniospassword
  2. Install Linux cifs utils
    sudo apt-get install  cifs-utils
    
  3. Create a mount directory
    mkdir /media/bobos
    
  4. Mount Windows shared directory to your local directory
    Mount the directory with read / write permissions. You can use either your domain as an address or you IP:

    sudo mount -t cifs //manios.mycompany.local/books /media/bobos/ -o username=manios,rw,password=maniospassword,domain=mydomain
    

    OR mount the directory with read only permissions:

    sudo mount -t cifs //manios.mycompany.local/books /media/bobos/ -o username=manios,password=maniospassword,domain=mydomain
    

    If the Windows host is not connected in an active directory domain and you log in with your local account, so you have to add sec=ntlm into -o parameter (according to this StackOverflow answer. So the command looks like this:

sudo mount -t cifs //manios.mycompany.local/books /media/bobos/ -o username=manios,password=maniospassword,sec=ntlm

Congratulations! You are complete! Now if you want to unmount the directory:

  • Run umount command
    umount /media/bobos/
    
  • Remove local directory
    rm -f /media/bobos/
    
  • I hope you find this guide useful!

    Posted in Uncategorized | Tagged , , , , , , , , , , | Leave a comment

    Fix Vmware tools ” kernel header path is not valid ” error

    UPDATE (2015-05-13): Confirming that the following approach works for Lubuntu versions 12.10, 14.04 and 14.10 and Linux Mint 15 and 17.

    Today I wanted to install VMware tools in a Linux Mint 15 and in a Lubuntu virtual machine. I followed the standard procedure as it is described in this VMWare knowledge base article. After setting gcc path I received the following error:

    run vmware-config-tools.pl, got same error message:**Searching for a valid kernel header path... The path "" is not valid. Would you like to change it?
    

    After an hour of searching and trying numerous solutions I found the solution
    in this AskUbuntu answer
    .So before executing sudo ./vmware-install.pl follow these steps:

    1. Download linux headers using the command
      sudo apt-get update && sudo apt-get install build-essential linux-headers-$(uname -r)
      
    2. Create a symbolic link to version.h file
      The vmware tools installer is looking for the version.h file in under [kernelsource path]/include/linux/version.h, it’s not there. The location of version.h is [kernelsource path]include/generated/uapi/linux/version.h

      sudo ln -s /usr/src/linux-headers-$(uname -r)/include/generated/uapi/linux/version.h /usr/src/linux-headers-$(uname -r)/include/linux/version.h
      

    Now you are able to install VMware tools without a problem! Good luck!

    Posted in Linux, Mint, Ubuntu, Virtualisation, vmware | Tagged , , , , , , , , | 24 Comments

    Vi set color scheme , line numbers and syntax highlighting

    Today I wanted to to set the Color Scheme in vi / vim in my Linux. The default configuration looks ugly:

    vi_colorscheme_default

    Default coloscheme for linux vi editor

    I wanted to look like this:

    vi_colorscheme_evening

    Evening colorcheme for linux vi editor

    So after some searching in the web I found an article in talkbinary and I did the following:
    1. Create in your home directory a file called .exrc

    touch ~/.exrc
    

    2. Open this file and insert the following parameters:

    set number
    syntax on
    colorscheme evening
    

    3. Save the file and you are done! Now you have a nice colourfull VI editor

    Posted in Linux, Mint, Raspbian, Ubuntu | Tagged , , , , , | Leave a comment