Daggy - Data Aggregation Utility
HomeGitHub
Search…
About Daggy
Getting Started
Data Aggregation Sources
Data Aggregation Snippets
Changelog
Powered By GitBook
Getting Started
Getting started data aggregation and streaming with daggy

Getting Daggy

Windows

Download installation package (exe) from last release page and install it

MacOS

1
brew install https://raw.githubusercontent.com/synacker/daggy_homebrew/master/daggy.rb
Copied!

Fedora

1
sudo dnf install daggy daggy-devel
Copied!

Linux

Download rpm/deb package from last release page and install it
All daggy dependencies (Qt5, libssh2, libyaml-cpp) are included into separate dir and does not effect system applications.

Build from source

Environment requirenments

​Conan, cmake, git and C++ compiler.

Build steps

1
git clone https://github.com/synacker/daggy.git
2
mkdir build
3
cd build
4
conan remote add bincrafters https://api.bintray.com/conan/bincrafters/public-conan
5
conan install ../daggy --build=missing
6
conan build ../daggy
7
cmake install
Copied!

Check installation

1
daggy --help
2
Usage: daggy [options] *.yaml|*.yml|*.json
3
​
4
Options:
5
-o, --output <folder> Set output folder
6
-f, --format <json|yaml> Source format
7
-i, --stdin Read data aggregation sources from stdin
8
-t, --timeout <time in ms> Auto complete timeout
9
-h, --help Displays help on commandline options.
10
--help-all Displays help including Qt specific options.
11
-v, --version Displays version information.
12
​
13
Arguments:
14
file data aggregation sources file
Copied!

Getting Started with data aggregation and streaming

Simple Data Aggregation Source

Create yaml or json file

YAML
JSON
simple.yaml
1
sources:
2
localhost:
3
type: local
4
commands:
5
pingYa:
6
exec: ping ya.ru
7
extension: log
Copied!
simple.json
1
{
2
"sources": {
3
"localhost": {
4
"commands": {
5
"pingYa": {
6
"extension": "log",
7
"exec": "ping ya.ru"
8
}
9
},
10
"type": "local"
11
}
12
}
13
}
Copied!

Run daggy

YAML
JSON
1
daggy simple.yaml
Copied!
1
daggy simple.json
Copied!

Check console output

1
23:07:23:977 | AppStat | Start aggregation in 01-04-20_23-07-23-977_simple
2
23:07:23:977 | ProvStat | localhost | New state: Started
3
23:07:23:977 | CommStat | localhost | pingYa | New state: Starting
4
23:07:23:977 | CommStat | localhost | pingYa | New state: Started
Copied!
There are all commands from simple.yaml/simple.json are streams in 01-04-20_23-07-23-977_simple with output files

Tailing streams from Simple Data Source

1
tail -f 01-04-20_23-07-23-977_simple/*
2
64 bytes from ya.ru (87.250.250.242): icmp_seq=99 ttl=249 time=21.2 ms
3
64 bytes from ya.ru (87.250.250.242): icmp_seq=100 ttl=249 time=18.8 ms
4
64 bytes from ya.ru (87.250.250.242): icmp_seq=101 ttl=249 time=23.5 ms
5
64 bytes from ya.ru (87.250.250.242): icmp_seq=102 ttl=249 time=18.8 ms
6
64 bytes from ya.ru (87.250.250.242): icmp_seq=103 ttl=249 time=18.8 ms
7
64 bytes from ya.ru (87.250.250.242): icmp_seq=104 ttl=249 time=17.4 ms
8
64 bytes from ya.ru (87.250.250.242): icmp_seq=105 ttl=249 time=17.4 ms
9
64 bytes from ya.ru (87.250.250.242): icmp_seq=106 ttl=249 time=20.1 ms
10
64 bytes from ya.ru (87.250.250.242): icmp_seq=107 ttl=249 time=25.8 ms
11
64 bytes from ya.ru (87.250.250.242): icmp_seq=108 ttl=249 time=35.1 ms
12
64 bytes from ya.ru (87.250.250.242): icmp_seq=109 ttl=249 time=21.1 ms
Copied!

Stop data aggregation and streaming

Type CTRL+C for stopping data aggregation and streaming. Type CTRL+C twice for hard stop application, without waiting cancelation of child local and remote processes.
1
23:07:23:977 | AppStat | Start aggregation in 01-04-20_23-07-23-977_simple
2
23:07:23:977 | ProvStat | localhost | New state: Started
3
23:07:23:977 | CommStat | localhost | pingYa | New state: Starting
4
23:07:23:977 | CommStat | localhost | pingYa | New state: Started
5
^C23:17:56:667 | ProvStat | localhost | New state: Finishing
6
23:17:56:668 | CommStat | localhost | pingYa | New state: Finished. Exit code: 0
7
23:17:56:668 | ProvStat | localhost | New state: Finished
8
23:17:56:668 | AppStat | Stop aggregation in 01-04-20_23-07-23-977_simple
Copied!

Investigate aggregated data

1
ls -l 01-04-20_23-07-23-977_simple/
2
-rw-r--r-- 1 muxa muxa 45574 апр 1 23:17 localhost-pingYa.log
Copied!

Example of Data Aggregation Sources with multiple commands and remote data aggregation and streaming

YAML
JSON
multiple_and_remote.yaml
1
aliases:
2
- &my_commands
3
pingYa:
4
exec: ping ya.ru
5
extension: log
6
pingGoo:
7
exec: ping goo.gl
8
extension: log
9
​
10
- &ssh_auth
11
user: muxa
12
key: /home/muxa/.ssh/id_rsa
13
​
14
sources:
15
localhost:
16
type: local
17
commands: *my_commands
18
remotehost:
19
host: 192.168.1.9
20
type: ssh2
21
parameters: *ssh_auth
22
commands: *my_commands
23
remotehost2:
24
host: 192.168.1.9
25
type: ssh2
26
parameters: *ssh_auth
27
commands: *my_commands
28
remotehost3:
29
host: 192.168.1.9
30
type: ssh2
31
parameters: *ssh_auth
32
commands: *my_commands
Copied!
multiple_and_remote.json
1
{
2
"sources": {
3
"remotehost3": {
4
"parameters": {
5
"user": "muxa",
6
"key": "/home/muxa/.ssh/id_rsa"
7
},
8
"commands": {
9
"pingYa": {
10
"extension": "log",
11
"exec": "ping ya.ru"
12
},
13
"pingGoo": {
14
"extension": "log",
15
"exec": "ping goo.gl"
16
}
17
},
18
"type": "ssh2",
19
"host": "192.168.1.9"
20
},
21
"remotehost2": {
22
"parameters": {
23
"user": "muxa",
24
"key": "/home/muxa/.ssh/id_rsa"
25
},
26
"commands": {
27
"pingYa": {
28
"extension": "log",
29
"exec": "ping ya.ru"
30
},
31
"pingGoo": {
32
"extension": "log",
33
"exec": "ping goo.gl"
34
}
35
},
36
"type": "ssh2",
37
"host": "192.168.1.9"
38
},
39
"remotehost": {
40
"parameters": {
41
"user": "muxa",
42
"key": "/home/muxa/.ssh/id_rsa"
43
},
44
"commands": {
45
"pingYa": {
46
"extension": "log",
47
"exec": "ping ya.ru"
48
},
49
"pingGoo": {
50
"extension": "log",
51
"exec": "ping goo.gl"
52
}
53
},
54
"type": "ssh2",
55
"host": "192.168.1.9"
56
},
57
"localhost": {
58
"commands": {
59
"pingYa": {
60
"extension": "log",
61
"exec": "ping ya.ru"
62
},
63
"pingGoo": {
64
"extension": "log",
65
"exec": "ping goo.gl"
66
}
67
},
68
"type": "local"
69
}
70
}
71
}
Copied!
Previous
About Daggy
Next
Data Aggregation Sources
Last modified 8mo ago
Copy link
Contents
Getting Daggy
Windows
MacOS
Fedora
Linux
Build from source
Getting Started with data aggregation and streaming
Simple Data Aggregation Source
Example of Data Aggregation Sources with multiple commands and remote data aggregation and streaming