Installing Storm and Kestrel

These examples document the original platform. Use your own deployment; this website does not provide the former hosted API or account service.

Steps to install Storm and Kestrel (contents of servioticy_queues.scala showed below)

STORM-local

#Install STORM 
##IMPORTANT: This is a non-distributed installation of STORM for testing purposes! No need 
wget http://apache.rediris.es/incubator/storm/apache-storm-0.9.1-incubating/apache-storm-0.9.1-incubating.tar.gz
rm -rf storm
rm -rf apache-storm-0.9.1-incubating
tar xzf apache-storm-0.9.1-incubating.tar.gz
mv apache-storm-0.9.1-incubating storm

#Install Kestrel
wget http://twitter.github.io/kestrel/download/kestrel-2.4.1.zip
rm -rf kestrel-2.4.1
unzip kestrel-2.4.1.zip
mv kestrel-2.4.1 kestrel
cp servioticy_queues.scala kestrel/config

Run Storm and Kestrel

runAllLocal

#Run Kestrel
rm -rf /tmp/kestrel-queue
rm -rf /tmp/kestrel.log
java -server -Xmx1024m -Dstage=servioticy_queues -jar kestrel/kestrel_2.9.2-2.4.1.jar &
sleep 20


#Run servIoTicy topology
CP=dispatcher-0.1-jar-with-dependencies.jar
for f in `ls storm/lib`
do
        CP=$CP":"storm-0.8.2/lib/$f

done

java -Dstorm.home=$PWD/storm -cp $CP dispatcher-0.1-jar-with-dependencies.jar com.servioticy.dispatcher.DispatcherTopology

Content of the servIoTicy Dispatcher component

dispatcher.xml

<context>
        <compoAPI>http://localhost</compoAPI>
        <kestrels>
                <kestrel>
                        <ip>localhost</ip>
                </kestrel>
                <port>2229</port>
        </kestrels>
</context>

Contents of servioticy_queues.scala:

servioticy_queues.scala

import com.twitter.conversions.storage._
import com.twitter.conversions.time._
import com.twitter.logging.config._
import com.twitter.ostrich.admin.config._
import net.lag.kestrel.config._

new KestrelConfig {
  listenAddress = "0.0.0.0"
  memcacheListenPort = 22133
  textListenPort = 2222
  thriftListenPort = 2229

  queuePath = "/tmp/kestrel-queue"
  clientTimeout = 30.seconds
  expirationTimerFrequency = 1.second

  maxOpenTransactions = 100

  // default queue settings:
  default.defaultJournalSize = 16.megabytes
  default.maxMemorySize = 128.megabytes
  default.maxJournalSize = 1.gigabyte
  default.maxItems = 0

  admin.httpPort = 2223

  admin.statsNodes = new StatsConfig {
    reporters = new TimeSeriesCollectorConfig
  }

  queues = new QueueBuilder {
    // keep items for no longer than a half hour, and don't accept any more if
    // the queue reaches 1.5M items.
    name = "avg"
    maxAge = 300.seconds
    maxItems = 1500000
  } :: new QueueBuilder {
    // don't keep a journal file for this queue. when kestrel exits, any
    // remaining contents will be lost.
    name = "transient_events"
    keepJournal = false
  } :: new QueueBuilder {
    name = "services"
    // Timeout for the descriptors
    maxAge = 300.seconds
    maxItems = 1500000
  }

  loggers = new LoggerConfig {
    level = Level.INFO
    handlers = new FileHandlerConfig {
      filename = "/tmp/kestrel.log"
      roll = Policy.Never
    }
  }

}