NAME
Vayne - Distribute task queue
SYNOPSIS
use Vayne;
use Vayne::Callback;
use Vayne::Tracker;
my @workload = <>; chomp @workload;
my $tracker = Vayne::Tracker->new();
my $step = Vayne->task('foo');
my $taskid = $tracker->add_task(
'region:region-first',
{
name => 'foo',
step => $step,
expire => 90,
},
@workload
);
my $call = Vayne::Callback->new();
my $stat = $call->wait($taskid);
GETTING STARTED
# First time only
> vayne-init -d $HOME/vayne
# Setup configurations in $HOME/vayne/conf (zookeeper mongodb redis)
# Add our first region, then the region info will upload to zk server.
> vayne-ctrl --set --region region-first --server redisserver:port --password redispasswd
# Define our first task, $HOME/vayne/task/foo
# check server ssh server
- name: 'check ssh' #step's name
worker: tcp #step's worker
param: #step's parameters
port: 22
input: ''
check: '^SSH-2.0-OpenSSH'
- name: 'foo'
worker: dump
param:
bar: baz
- name: 'only suc'
need:
- 'check ssh': 1
worker: dump
param:
- array
- key1: value1
key2: value2
# tracke the job result
- name: 'tracker'
worker: track
# Switch the server you run workers to our first region.
> vayne-ctrl --switch --region region-first
# Run workers.
> $HOME/vayne/worker/tcp &
> $HOME/vayne/worker/dump &
> $HOME/vayne/worker/tracker &
# Run task tracker.
> vayne-tracker &
# Submit our task by CLI.
> echo '127.0.0.1'|vayne-task --add --name foo --expire 60 --strategy region:region-first
# or
> vayne-task --add --name foo --expire 60 --strategy region:region-first < server_list
# Query our task through taskid by CLI.
> vayne-task --taskid 9789F5E6-2644-11E6-A6F0-AF9AF8F9E07F --query
# Or use Vayne lib in your program like SYNOPSIS.
DESCRIPTION
Vayne is a distribute task queue with many feature.
FEATURE
Logical Region with Flexible Spawning Strategy
Has the concept of logical region. You can spawn task into different
region with strategy. Spawning strategy can be easily write.
Custome Task Flow with Reusable Worker
Worker is a process can do some specific stuffs. Step have a step's
name, a worker's name and parameters. You can define custome task
flow by constructing any steps.
Simple Worker Interface with Good Performance
Vayne::Worker is using Coro which can provide excellent performance
in network IO. Worker has a simple interface to write, also you can
use Coro::* module to enhance worker performance. Whole system is
combined with Message Queue. You can get a better performance easily
by increasing the worker counts while MQ is not the bottleneck.
HOW IT WORKS
+--------+
|Worker A| x N
+--------+
+------------+ Workers may run on several servers
| | +--------+
| Region A | |Worker B| x N
| | +--------+
+------------+
.....
+-----------+ +----------+
| | +------------+ |JobTracker| x N
| Task Conf | | | +----------+
| | | Region B |
| +-------+ | | |
| | step1 | | +-----------+ +------------+ +-----------+ +-----------+
| +-------+ | | | Spawn Jobs Save Job Information | | +------> | |
| | + | workloads | +----------------> +---------------------> | Mongodb | |TaskTracker|
| +-------+ | | | with Strategy to Center Mongodb | | <------+ | |
| | step2 | | +-----------+ +------------+ +-----------+ +-----------+
| +-------+ | | |
| ... | | Region C | ..... ^
| +-------+ | | | |
| | stepN | | +------------+ |
| +-------+ | |
| | |
+-----------+ +------------+ |
| | |
| Region D | |
| | | |
| +------------+ |
| |
| |
| |
| |
| Save Task Information to Center Mongodb |
+----------------------------------------------------------------------------------------------------------+
0. Task Conf & Step
The Task Conf is combined with several ordered steps. Each step have a
step's name, a worker's name and parameters. Workload will be prosessed
step by step.
1. Spawn Task
Vayne support CLI and API to spawn a task. A task contain numbers of
jobs. Task info will write to task collection in mongodb first. Jobs
will be hashed into saperated region by strategy. Then enqueue jobs to
their region's redis queue named by first step of the job, and write to
job collection in mongodb.
2. Queue & Region
Like Redis::JobQueue, Vayne use redis for job queuing and job info
caching. The data structure is nearly the same as "JobQueue data
structure stored in Redis" in Redis::JobQueue.
Each region has a queue(redis server). Both their infomation are saved
on zookeeper server.
Each real server which you want to run workers should belong to a
region.
Worker will register its names under real server's region on zookeeper
server when it start.
Details see "DATA STRUCTURE" in Vayne::Zk.
2. Worker
When it start, worker register its names on zookeeper server. Then
generate some Coro threads below:
Check the Registration
Go die when the registration changed. Ex: Region info changed; Real
Server switch to another region; Connection to zk failed.
* The worker will die very quickly when zookeeper server is not
available. It may cause some problems. Should be careful.
Job Consumer
BLPOP queues which worker registered, then put the job into
Coro::Channel
Worker
Get job from Coro::Channel, and do the stuff with it. Tag the result
and status on the job. Put the job to update Coro::Channel.
Update Job Info
Get Job from update Coro::Channel. Push the job to next queue
according to the job's step.
INT, TERM, HUP signals will be catched. Then graceful stop the worker.
4. JOB TRACKER
Job tracker is a special worker, it just send the job info dealed by
previous workers to mongodb. Usually 'tracker' should be the last step
of a job.
5. TASK TRACKER
Script vayne-tracker Loop bla..
BACKEND
Redis-3.2 http://redis.io/
Zookeeper-3.3.6 http://zookeeper.apache.org/
MongoDB-3.0.6 https://www.mongodb.com/
DATA STRUCTURE
Zookeeper
"DATA STRUCTURE" in Vayne::Zk
Redis
Data Structure for job&queue is nearly the same as "JobQueue data
structure stored in Redis" in Redis::JobQueue
MongoDB
bla bla..
HOW TO WRITE A WORKER
bla bla..
AUTHOR
SiYu Zhao <zuyis@cpan.org>
COPYRIGHT
Copyright 2016- SiYu Zhao
LICENSE
This library is free software; you can redistribute it and/or modify it
under the same terms as Perl itself.
SEE ALSO
Redis::JobQueue