PG1X

My Knowledge Base

User Tools

Site Tools


tech:network:netflow:elastiflow:elastiflow

ElastiFlow

Ubuntu Server Bionic Static IP Configuration

Virtual Machine Configuration Overview

Almost OS Installation process snipped.

Ubuntu Networking

Networking configuration task done via Netplan abstraction framework.

sudo mv /etc/netplan/50-cloud-init.yaml /etc/netplan/01-netcfg.yaml
sudo vim /etc/netplan/01-netcfg.yaml
/etc/netplan/01-netcfg.yaml
network:
 version: 2
 renderer: networkd
 ethernets:
   ens33:
     dhcp4: no
     dhcp6: no
     addresses: [ 172.16.2.222/24 ]
     gateway4: 172.16.2.1
     nameservers:
       addresses: [ 8.8.8.8, 8.8.4.4, 1.1.1.1 ]
sudo netplan apply

Check systemd-resolved naming service.

$ sudo systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (ens33)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.8.8
                      8.8.4.4
                      1.1.1.1

Installation

sudo add-apt-repository -y ppa:webupd8team/java
sudo apt update
sudo apt install -y oracle-java8-installer
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt install -y apt-transport-https
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
sudo apt update
sudo apt -y install elasticsearch
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
sudo apt update
sudo apt -y install kibana
sudo systemctl daemon-reload
sudo systemctl enable kibana.service
sudo systemctl start kibana.service
sudo apt update
sudo apt -y install logstash

Restrict access to Elasticsearch allow from localhost only.

sudo cp -p /etc/elasticsearch/elasticsearch.yml /root/
sudo vim /etc/elasticsearch/elasticsearch.yml
/etc/elasticsearch/elasticsearch.yml
network.host: localhost
wnoguchi@elastiflow:~$ sudo diff -u /root/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml
--- /root/elasticsearch.yml     2018-06-11 23:44:16.000000000 +0000
+++ /etc/elasticsearch/elasticsearch.yml        2018-06-27 21:24:48.691357395 +0000
@@ -52,7 +52,7 @@
 #
 # Set the bind address to a specific IP (IPv4 or IPv6):
 #
-#network.host: 192.168.0.1
+network.host: localhost
 #
 # Set a custom port for HTTP:
 #
wnoguchi@elastiflow:~$ curl localhost:9200/?pretty
{
  "name" : "u0wtgQd",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "kMI9m6FkS3ac1j9ojaMvEQ",
  "version" : {
    "number" : "6.3.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "424e937",
    "build_date" : "2018-06-11T23:38:03.357887Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
sudo systemctl restart elasticsearch.service
wnoguchi@elastiflow:~$ curl localhost:9200/?pretty
curl: (7) Failed to connect to localhost port 9200: Connection refused
wnoguchi@elastiflow:~$ curl localhost:9200/?pretty
{
  "name" : "u0wtgQd",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "kMI9m6FkS3ac1j9ojaMvEQ",
  "version" : {
    "number" : "6.3.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "424e937",
    "build_date" : "2018-06-11T23:38:03.357887Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
sudo cp -p /etc/kibana/kibana.yml /root/
sudo vim /etc/kibana/kibana.yml
/etc/kibana/kibana.yml
server.host: 0.0.0.0
wnoguchi@elastiflow:~$ sudo diff -u /root/kibana.yml /etc/kibana/kibana.yml
--- /root/kibana.yml    2018-06-27 21:51:44.394821172 +0000
+++ /etc/kibana/kibana.yml      2018-06-27 21:52:14.998766452 +0000
@@ -4,7 +4,7 @@
 # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
 # The default is 'localhost', which usually means remote machines will not be able to connect.
 # To allow connections from remote users, set this parameter to a non-loopback address.
-#server.host: "localhost"
+server.host: 0.0.0.0
 
 # Enables you to specify a path to mount Kibana at if you are running behind a proxy.
 # Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
sudo systemctl restart kibana.service

http://172.16.2.222:5601

Configure Cisco NetFlow

Install Route to Cisco Equipment Loopback address Lo0

IOS15 Box

sudo ip route add 172.16.255.1/32 via 172.16.2.1

IOS12 Box

sudo ip route add 172.16.255.2/32 via 172.16.2.2
/etc/logstash/jvm.options
#-Xms1g
#-Xmx1g
-Xms4g
-Xmx4g

Configure ELK

sudo /usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow
sudo /usr/share/logstash/bin/logstash-plugin update logstash-codec-netflow
sudo /usr/share/logstash/bin/logstash-plugin update logstash-input-udp
sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-dns
wnoguchi@elastiflow:~$ sudo /usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow
Validating logstash-codec-sflow
Installing logstash-codec-sflow
Installation successful
wnoguchi@elastiflow:~$ sudo /usr/share/logstash/bin/logstash-plugin update logstash-codec-netflow
Updating logstash-codec-netflow
Updated logstash-codec-netflow 3.12.0 to 4.0.2
wnoguchi@elastiflow:~$ sudo /usr/share/logstash/bin/logstash-plugin update logstash-input-udp
Updating logstash-input-udp
Updated logstash-input-udp 3.3.2 to 3.3.3
wnoguchi@elastiflow:~$ sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-dns
Updating logstash-filter-dns
Updated logstash-filter-dns 3.0.9 to 3.0.11
mkdir flowtemp
cd flowtemp/
wget https://github.com/robcowart/elastiflow/archive/master.zip
sudo apt install -y unzip
unzip master.zip
wnoguchi@elastiflow:~/flowtemp$ unzip master.zip
Archive:  master.zip
9df91682b391432d557cfc31a7022e15d348586f
   creating: elastiflow-master/
  inflating: elastiflow-master/LICENSE.md
  inflating: elastiflow-master/README.md
   creating: elastiflow-master/kibana/
  inflating: elastiflow-master/kibana/elastiflow.dashboards.6.2.x.json
  inflating: elastiflow-master/kibana/elastiflow.dashboards.6.3.x.json
  inflating: elastiflow-master/kibana/elastiflow.index_pattern.json
   creating: elastiflow-master/logstash.service.d/
  inflating: elastiflow-master/logstash.service.d/elastiflow.conf
   creating: elastiflow-master/logstash/
   creating: elastiflow-master/logstash/elastiflow/
   creating: elastiflow-master/logstash/elastiflow/conf.d/
  inflating: elastiflow-master/logstash/elastiflow/conf.d/10_input_ipfix_ipv4.logstash.conf
  inflating: elastiflow-master/logstash/elastiflow/conf.d/10_input_ipfix_ipv6.logstash.conf.disabled
  inflating: elastiflow-master/logstash/elastiflow/conf.d/10_input_netflow_ipv4.logstash.conf
  inflating: elastiflow-master/logstash/elastiflow/conf.d/10_input_netflow_ipv6.logstash.conf.disabled
  inflating: elastiflow-master/logstash/elastiflow/conf.d/10_input_sflow_ipv4.logstash.conf
  inflating: elastiflow-master/logstash/elastiflow/conf.d/10_input_sflow_ipv6.logstash.conf.disabled
  inflating: elastiflow-master/logstash/elastiflow/conf.d/20_filter_10_begin.logstash.conf
  inflating: elastiflow-master/logstash/elastiflow/conf.d/20_filter_20_netflow.logstash.conf
  inflating: elastiflow-master/logstash/elastiflow/conf.d/20_filter_30_ipfix.logstash.conf
  inflating: elastiflow-master/logstash/elastiflow/conf.d/20_filter_40_sflow.logstash.conf
  inflating: elastiflow-master/logstash/elastiflow/conf.d/20_filter_90_post_process.logstash.conf
  inflating: elastiflow-master/logstash/elastiflow/conf.d/30_output.logstash.conf
   creating: elastiflow-master/logstash/elastiflow/dictionaries/
  inflating: elastiflow-master/logstash/elastiflow/dictionaries/app_id.srctype.yml
  inflating: elastiflow-master/logstash/elastiflow/dictionaries/app_id.yml
  inflating: elastiflow-master/logstash/elastiflow/dictionaries/iana_protocol_numbers.yml
  inflating: elastiflow-master/logstash/elastiflow/dictionaries/iana_service_names_dccp.yml
  inflating: elastiflow-master/logstash/elastiflow/dictionaries/iana_service_names_sctp.yml
  inflating: elastiflow-master/logstash/elastiflow/dictionaries/iana_service_names_tcp.yml
  inflating: elastiflow-master/logstash/elastiflow/dictionaries/iana_service_names_udp.yml
  inflating: elastiflow-master/logstash/elastiflow/dictionaries/ip_rep_basic.yml
  inflating: elastiflow-master/logstash/elastiflow/dictionaries/sflow_header_protocol.yml
  inflating: elastiflow-master/logstash/elastiflow/dictionaries/sflow_source_id_type.yml
  inflating: elastiflow-master/logstash/elastiflow/dictionaries/tcp_flags.yml
   creating: elastiflow-master/logstash/elastiflow/geoipdbs/
 extracting: elastiflow-master/logstash/elastiflow/geoipdbs/COPYRIGHT.txt
  inflating: elastiflow-master/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb
  inflating: elastiflow-master/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb
  inflating: elastiflow-master/logstash/elastiflow/geoipdbs/LICENSE.txt
   creating: elastiflow-master/logstash/elastiflow/templates/
  inflating: elastiflow-master/logstash/elastiflow/templates/elastiflow.template.json
   creating: elastiflow-master/profile.d/
  inflating: elastiflow-master/profile.d/elastiflow.sh
sudo cp -a elastiflow-master/logstash/elastiflow/. /etc/logstash/elastiflow/
sudo cp -a elastiflow-master/logstash.service.d/. /etc/systemd/system/logstash.service.d/
/etc/logstash/pipelines.yml
- pipeline.id: elastiflow
  path.config: "/etc/logstash/elastiflow/conf.d/*.conf"

Keep NetFlow Default Port and Listen all address.

/etc/systemd/system/logstash.service.d/elastiflow.conf
# Netflow - IPv4
Environment="ELASTIFLOW_NETFLOW_IPV4_HOST=0.0.0.0"
Environment="ELASTIFLOW_NETFLOW_IPV4_PORT=2055"

Make sure open your incoming NetFlow data port.

wnoguchi@elastiflow:~/flowtemp$ sudo ufw status
Status: inactive
wnoguchi@elastiflow:~/flowtemp$ sudo systemctl is-active logstash
inactive
wnoguchi@elastiflow:~/flowtemp$ sudo systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/logstash.service.d
           └─elastiflow.conf
   Active: inactive (dead)
sudo /usr/share/logstash/bin/system-install
sudo systemctl daemon-reload
sudo systemctl enable logstash
sudo systemctl start logstash
sudo systemctl is-active logstash
sudo systemctl status logstash
wnoguchi@elastiflow:~/flowtemp$ sudo /usr/share/logstash/bin/system-install
Successfully created system startup script for Logstash
wnoguchi@elastiflow:~/flowtemp$ sudo systemctl daemon-reload
wnoguchi@elastiflow:~/flowtemp$ sudo systemctl enable logstash
Created symlink /etc/systemd/system/multi-user.target.wants/logstash.service → /etc/systemd/system/logstash.service.
wnoguchi@elastiflow:~/flowtemp$ sudo systemctl start logstash
wnoguchi@elastiflow:~/flowtemp$ sudo systemctl is-active logstash
active
wnoguchi@elastiflow:~/flowtemp$ sudo systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/logstash.service.d
           └─elastiflow.conf
   Active: active (running) since Sun 2018-07-01 10:25:40 UTC; 8s ago
 Main PID: 5294 (java)
    Tasks: 24 (limit: 19150)
   CGroup: /system.slice/logstash.service
           └─5294 /usr/bin/java -Xms4g -Xmx4g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupanc

Jul 01 10:25:40 elastiflow systemd[1]: Started logstash.
wnoguchi@elastiflow:~/flowtemp$ tail -f /var/log/logstash/logstash-plain.log
[2018-07-01T10:26:08,601][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2018-07-01T10:26:08,610][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[2018-07-01T10:26:09,179][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"e486d6cb-7e99-4320-936a-212804a8cd16", :path=>"/var/lib/logstash/uuid"}
[2018-07-01T10:26:09,940][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.0"}
[2018-07-01T10:26:10,132][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
[2018-07-01T10:28:50,517][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"elastiflow", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-07-01T10:28:51,145][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elastic:xxxxxx@127.0.0.1:9200/]}}
[2018-07-01T10:28:51,176][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elastic:xxxxxx@127.0.0.1:9200/, :path=>"/"}
[2018-07-01T10:28:51,665][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elastic:xxxxxx@127.0.0.1:9200/"}
[2018-07-01T10:28:51,768][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2018-07-01T10:28:51,774][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2018-07-01T10:28:51,800][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>"/etc/logstash/elastiflow/templates/elastiflow.template.json"}
[2018-07-01T10:28:51,862][INFO ][logstash.outputs.elasticsearch] flow.xlate_src_addr_ipv4", "mapping"=>{"type"=>"ip"}}}, {"netflow.xlate_src_addr_ipv6"=>{"path_match"=>"netflow.xlate_src_addr_ipv6", "mapping"=>{"type"=>"ip"}}}, {"netflow.xlate_src_port"=>{"path_match"=>"netflow.xlate_src_port", "mapping"=>{"type"=>"long"}}}, {"sflow.agent_ip"=>{"path_match"=>"sflow.agent_ip", "mapping"=>{"type"=>"ip"}}}, {"sflow.drops"=>{"path_match"=>"sflow.drops", "mapping"=>{"type"=>"long"}}}, {"sflow.dst_ip"=>{"path_match"=>"sflow.dst_ip", "mapping"=>{"type"=>"ip"}}}, {"sflow.dst_mac"=>{"path_match"=>"sflow.dst_mac", "mapping"=>{"type"=>"keyword"}}}, {"sflow.dst_mask_len"=>{"path_match"=>"sflow.dst_mask_len", "mapping"=>{"type"=>"long"}}}, {"sflow.dst_port"=>{"path_match"=>"sflow.dst_port", "mapping"=>{"type"=>"long"}}}, {"sflow.dst_priority"=>{"path_match"=>"sflow.dst_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.dst_vlan"=>{"path_match"=>"sflow.dst_vlan", "mapping"=>{"type"=>"long"}}}, {"sflow.eth_dst"=>{"path_match"=>"sflow.eth_dst", "mapping"=>{"type"=>"keyword"}}}, {"sflow.eth_src"=>{"path_match"=>"sflow.eth_src", "mapping"=>{"type"=>"keyword"}}}, {"sflow.eth_type"=>{"path_match"=>"sflow.eth_type", "mapping"=>{"type"=>"long"}}}, {"sflow.flow_sequence_number"=>{"path_match"=>"sflow.flow_sequence_number", "mapping"=>{"type"=>"long"}}}, {"sflow.frame_length"=>{"path_match"=>"sflow.frame_length", "mapping"=>{"type"=>"long"}}}, {"sflow.frame_length_times_sampling_rate"=>{"path_match"=>"sflow.frame_length_times_sampling_rate", "mapping"=>{"type"=>"long"}}}, {"sflow.header_size"=>{"path_match"=>"sflow.header_size", "mapping"=>{"type"=>"long"}}}, {"sflow.input_interface"=>{"path_match"=>"sflow.input_interface", "mapping"=>{"type"=>"long"}}}, {"sflow.input_interface_format"=>{"path_match"=>"sflow.input_interface_format", "mapping"=>{"type"=>"long"}}}, {"sflow.input_interface_value"=>{"path_match"=>"sflow.input_interface_value", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_address_next_hop_router"=>{"path_match"=>"sflow.ip_address_next_hop_router", "mapping"=>{"type"=>"ip"}}}, {"sflow.ip_checksum"=>{"path_match"=>"sflow.ip_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_dscp"=>{"path_match"=>"sflow.ip_dscp", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_ecn"=>{"path_match"=>"sflow.ip_ecn", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_flags"=>{"path_match"=>"sflow.ip_flags", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_fragment_offset"=>{"path_match"=>"sflow.ip_fragment_offset", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_header_length"=>{"path_match"=>"sflow.ip_header_length", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_identification"=>{"path_match"=>"sflow.ip_identification", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_next_header"=>{"path_match"=>"sflow.ip_next_header", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_packet_length"=>{"path_match"=>"sflow.ip_packet_length", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_priority"=>{"path_match"=>"sflow.ip_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_protocol"=>{"path_match"=>"sflow.ip_protocol", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_total_length"=>{"path_match"=>"sflow.ip_total_length", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_ttl"=>{"path_match"=>"sflow.ip_ttl", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_type"=>{"path_match"=>"sflow.ip_type", "mapping"=>{"type"=>"long"}}}, {"sflow.ip_version"=>{"path_match"=>"sflow.ip_version", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface"=>{"path_match"=>"sflow.output_interface", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface_format"=>{"path_match"=>"sflow.output_interface_format", "mapping"=>{"type"=>"long"}}}, {"sflow.output_interface_value"=>{"path_match"=>"sflow.output_interface_value", "mapping"=>{"type"=>"long"}}}, {"sflow.packet_length"=>{"path_match"=>"sflow.packet_length", "mapping"=>{"type"=>"long"}}}, {"sflow.padded"=>{"path_match"=>"sflow.padded", "mapping"=>{"type"=>"long"}}}, {"sflow.protocol"=>{"path_match"=>"sflow.protocol", "mapping"=>{"type"=>"keyword"}}}, {"sflow.sample_length"=>{"path_match"=>"sflow.sample_length", "mapping"=>{"type"=>"long"}}}, {"sflow.sample_pool"=>{"path_match"=>"sflow.sample_pool", "mapping"=>{"type"=>"long"}}}, {"sflow.sample_seq_number"=>{"path_match"=>"sflow.sample_seq_number", "mapping"=>{"type"=>"long"}}}, {"sflow.sampling_rate"=>{"path_match"=>"sflow.sampling_rate", "mapping"=>{"type"=>"long"}}}, {"sflow.sflow_type"=>{"path_match"=>"sflow.sflow_type", "mapping"=>{"type"=>"keyword"}}}, {"sflow.sflow_version"=>{"path_match"=>"sflow.sflow_version", "mapping"=>{"type"=>"long"}}}, {"sflow.size_header"=>{"path_match"=>"sflow.size_header", "mapping"=>{"type"=>"long"}}}, {"sflow.source_id_index"=>{"path_match"=>"sflow.source_id_index", "mapping"=>{"type"=>"long"}}}, {"sflow.source_id_index_name"=>{"path_match"=>"sflow.source_id_index_name", "mapping"=>{"type"=>"keyword"}}}, {"sflow.source_id_type"=>{"path_match"=>"sflow.source_id_type", "mapping"=>{"type"=>"keyword"}}}, {"sflow.src_ip"=>{"path_match"=>"sflow.src_ip", "mapping"=>{"type"=>"ip"}}}, {"sflow.src_mac"=>{"path_match"=>"sflow.src_mac", "mapping"=>{"type"=>"keyword"}}}, {"sflow.src_mask_len"=>{"path_match"=>"sflow.src_mask_len", "mapping"=>{"type"=>"long"}}}, {"sflow.src_port"=>{"path_match"=>"sflow.src_port", "mapping"=>{"type"=>"long"}}}, {"sflow.src_priority"=>{"path_match"=>"sflow.src_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.src_vlan"=>{"path_match"=>"sflow.src_vlan", "mapping"=>{"type"=>"long"}}}, {"sflow.stripped"=>{"path_match"=>"sflow.stripped", "mapping"=>{"type"=>"long"}}}, {"sflow.sub_agent_id"=>{"path_match"=>"sflow.sub_agent_id", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_ack_number"=>{"path_match"=>"sflow.tcp_ack_number", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_checksum"=>{"path_match"=>"sflow.tcp_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_flags"=>{"path_match"=>"sflow.tcp_flags", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_header_length"=>{"path_match"=>"sflow.tcp_header_length", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_reserved"=>{"path_match"=>"sflow.tcp_reserved", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_seq_number"=>{"path_match"=>"sflow.tcp_seq_number", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_urgent_pointer"=>{"path_match"=>"sflow.tcp_urgent_pointer", "mapping"=>{"type"=>"long"}}}, {"sflow.tcp_window_size"=>{"path_match"=>"sflow.tcp_window_size", "mapping"=>{"type"=>"long"}}}, {"sflow.udp_checksum"=>{"path_match"=>"sflow.udp_checksum", "mapping"=>{"type"=>"long"}}}, {"sflow.udp_length"=>{"path_match"=>"sflow.udp_length", "mapping"=>{"type"=>"long"}}}, {"sflow.uptime_in_ms"=>{"path_match"=>"sflow.uptime_in_ms", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_cfi"=>{"path_match"=>"sflow.vlan_cfi", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_id"=>{"path_match"=>"sflow.vlan_id", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_priority"=>{"path_match"=>"sflow.vlan_priority", "mapping"=>{"type"=>"long"}}}, {"sflow.vlan_type"=>{"path_match"=>"sflow.vlan_type", "mapping"=>{"type"=>"long"}}}, {"string_fields"=>{"mapping"=>{"type"=>"keyword"}, "match_mapping_type"=>"string", "match"=>"*"}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "event"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"host"=>{"type"=>"keyword"}, "type"=>{"type"=>"keyword"}}}, "flow"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"application"=>{"type"=>"keyword"}, "autonomous_system"=>{"type"=>"keyword"}, "bgp_next_hop"=>{"type"=>"ip"}, "bgp_valid_state"=>{"type"=>"long"}, "bytes"=>{"type"=>"long"}, "city"=>{"type"=>"keyword"}, "client_addr"=>{"type"=>"ip"}, "client_asn"=>{"type"=>"long"}, "client_autonomous_system"=>{"type"=>"keyword"}, "client_city"=>{"type"=>"keyword"}, "client_country"=>{"type"=>"keyword"}, "client_country_code"=>{"type"=>"keyword"}, "client_geo_location"=>{"type"=>"geo_point"}, "client_hostname"=>{"type"=>"keyword"}, "country"=>{"type"=>"keyword"}, "country_code"=>{"type"=>"keyword"}, "direction"=>{"type"=>"keyword"}, "dst_addr"=>{"type"=>"ip"}, "dst_addr_trans"=>{"type"=>"ip"}, "dst_asn"=>{"type"=>"long"}, "dst_autonomous_system"=>{"type"=>"keyword"}, "dst_city"=>{"type"=>"keyword"}, "dst_country"=>{"type"=>"keyword"}, "dst_country_code"=>{"type"=>"keyword"}, "dst_geo_location"=>{"type"=>"geo_point"}, "dst_hostname"=>{"type"=>"keyword"}, "dst_mac"=>{"type"=>"keyword"}, "dst_mask_len"=>{"type"=>"long"}, "dst_port"=>{"type"=>"long"}, "dst_port_trans"=>{"type"=>"long"}, "dst_port_name"=>{"type"=>"keyword"}, "dst_rep_tags"=>{"type"=>"keyword"}, "input_snmp"=>{"type"=>"keyword"}, "ip_protocol"=>{"type"=>"keyword"}, "ip_version"=>{"type"=>"keyword"}, "next_hop"=>{"type"=>"ip"}, "output_snmp"=>{"type"=>"keyword"}, "packets"=>{"type"=>"long"}, "rep_tags"=>{"type"=>"keyword"}, "sampling_interval"=>{"type"=>"long"}, "server_addr"=>{"type"=>"ip"}, "server_asn"=>{"type"=>"long"}, "server_autonomous_system"=>{"type"=>"keyword"}, "server_city"=>{"type"=>"keyword"}, "server_country"=>{"type"=>"keyword"}, "server_country_code"=>{"type"=>"keyword"}, "server_geo_location"=>{"type"=>"geo_point"}, "server_hostname"=>{"type"=>"keyword"}, "service_name"=>{"type"=>"keyword"}, "service_port"=>{"type"=>"long"}, "src_addr"=>{"type"=>"ip"}, "src_addr_trans"=>{"type"=>"ip"}, "src_asn"=>{"type"=>"long"}, "src_autonomous_system"=>{"type"=>"keyword"}, "src_city"=>{"type"=>"keyword"}, "src_country"=>{"type"=>"keyword"}, "src_country_code"=>{"type"=>"keyword"}, "src_geo_location"=>{"type"=>"geo_point"}, "src_hostname"=>{"type"=>"keyword"}, "src_mac"=>{"type"=>"keyword"}, "src_mask_len"=>{"type"=>"long"}, "src_port"=>{"type"=>"long"}, "src_port_trans"=>{"type"=>"long"}, "src_port_name"=>{"type"=>"keyword"}, "src_rep_tags"=>{"type"=>"keyword"}, "tcp_flags"=>{"type"=>"keyword"}, "tos"=>{"type"=>"long"}, "traffic_direction"=>{"type"=>"keyword"}, "traffic_locality"=>{"type"=>"keyword"}, "vlan"=>{"type"=>"long"}}}, "node"=>{"dynamic"=>true, "type"=>"object", "properties"=>{"ipaddr"=>{"type"=>"ip"}, "hostname"=>{"type"=>"keyword"}}}, "tags"=>{"type"=>"keyword"}}}}}}
[2018-07-01T10:28:51,954][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/elastiflow-3.1.0
[2018-07-01T10:28:52,357][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]}
[2018-07-01T10:28:52,787][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"}
[2018-07-01T10:28:52,814][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"}
[2018-07-01T10:29:01,312][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-City.mmdb"}
[2018-07-01T10:29:01,314][INFO ][logstash.filters.geoip   ] Using geoip database {:path=>"/etc/logstash/elastiflow/geoipdbs/GeoLite2-ASN.mmdb"}
[2018-07-01T10:29:10,753][INFO ][logstash.inputs.tcp      ] Starting tcp input listener {:address=>"0.0.0.0:4739", :ssl_enable=>"false"}
[2018-07-01T10:29:11,057][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"elastiflow", :thread=>"#<Thread:0x7118c33b run>"}
[2018-07-01T10:29:11,062][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:2055"}
[2018-07-01T10:29:11,062][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:4739"}
[2018-07-01T10:29:11,082][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:6343"}
[2018-07-01T10:29:11,155][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:2055", :receive_buffer_bytes=>"106496", :queue_size=>"4096"}
[2018-07-01T10:29:11,155][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:4739", :receive_buffer_bytes=>"106496", :queue_size=>"4096"}
[2018-07-01T10:29:11,176][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:6343", :receive_buffer_bytes=>"106496", :queue_size=>"4096"}
[2018-07-01T10:29:11,267][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:elastiflow], :non_running_pipelines=>[]}
[2018-07-01T10:29:11,602][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
wnoguchi@elastiflow:~/flowtemp$ sudo tcpdump -n port 2055
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes
10:33:18.525255 IP 172.16.2.1.50164 > 172.16.2.222.2055: UDP, length 1464
10:33:22.518180 IP 172.16.2.1.50164 > 172.16.2.222.2055: UDP, length 1464
10:33:33.535180 IP 172.16.2.1.50164 > 172.16.2.222.2055: UDP, length 1176
10:33:46.535861 IP 172.16.2.1.50164 > 172.16.2.222.2055: UDP, length 888
10:33:57.518165 IP 172.16.2.1.50164 > 172.16.2.222.2055: UDP, length 1464
10:34:03.525935 IP 172.16.2.1.50164 > 172.16.2.222.2055: UDP, length 1464
10:34:07.538476 IP 172.16.2.1.50164 > 172.16.2.222.2055: UDP, length 1464
10:34:18.537847 IP 172.16.2.1.50164 > 172.16.2.222.2055: UDP, length 1320
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel
wnoguchi@elastiflow:~/flowtemp$ sudo tcpdump -n port 2055 -w netflow-201807011935.pcap
tcpdump: listening on ens33, link-type EN10MB (Ethernet), capture size 262144 bytes
^C243 packets captured
243 packets received by filter
0 packets dropped by kernel

Configure Kibana

Let's Visualize.

Make sure in flowtemp directory

wnoguchi@elastiflow:~$ ls
flowtemp
wnoguchi@elastiflow:~$ cd flowtemp/
wnoguchi@elastiflow:~/flowtemp$ ls
elastiflow-master  master.zip  netflow-201807011935.pcap
wnoguchi@elastiflow:~/flowtemp$ pwd
/home/wnoguchi/flowtemp

Import ElastiFlow Indexes.

curl -X POST http://172.16.2.222:5601/api/saved_objects/index-pattern/elastiflow-* \
  -H "Content-Type: application/json" \
  -H "kbn-xsrf: true" \
  -d @elastiflow-master/kibana/elastiflow.index_pattern.json
wnoguchi@elastiflow:~/flowtemp$ curl -X POST http://172.16.2.222:5601/api/saved_objects/index-pattern/elastiflow-* \
>   -H "Content-Type: application/json" \
>   -H "kbn-xsrf: true" \
>   -d @elastiflow-master/kibana/elastiflow.index_pattern.json
{"id":"elastiflow-*","type":"index-pattern","updated_at":"2018-07-01T10:52:06.640Z","version":1,"attributes":{"title":"elastiflow-*","timeFieldName":"@timestamp","fields":"[{\"name\":\"@timestamp\",\"type\":\"date\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"@version\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"_id\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_index\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_score\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_source\",\"type\":\"_source\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"event.host\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"event.type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.application\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.autonomous_system\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.bgp_next_hop\",\"type\":\"ip\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.bgp_valid_state\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.bytes\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.city\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.client_addr\",\"type\":\"ip\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.client_asn\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.client_autonomous_system\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.client_city\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.client_country\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.client_country_code\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.client_geo_location\",\"type\":\"geo_point\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.client_hostname\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.country\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.country_code\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.direction\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_addr\",\"type\":\"ip\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_addr_trans\",\"type\":\"ip\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_asn\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_autonomous_system\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_city\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_country\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_country_code\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_geo_location\",\"type\":\"geo_point\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_hostname\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_mac\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_mask_len\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_port\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_port_name\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_port_trans\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.dst_rep_tags\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.input_snmp\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.ip_protocol\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.ip_version\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.next_hop\",\"type\":\"ip\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.output_snmp\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.packets\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.rep_tags\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.sampling_interval\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.server_addr\",\"type\":\"ip\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.server_asn\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.server_autonomous_system\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.server_city\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.server_country\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.server_country_code\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.server_geo_location\",\"type\":\"geo_point\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.server_hostname\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.service_name\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.service_port\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_addr\",\"type\":\"ip\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_addr_trans\",\"type\":\"ip\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_asn\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_autonomous_system\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_city\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_country\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_country_code\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_geo_location\",\"type\":\"geo_point\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_hostname\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_mac\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_mask_len\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_port\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_port_name\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_port_trans\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.src_rep_tags\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.tcp_flags\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.tos\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.traffic_direction\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.traffic_locality\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"flow.vlan\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ipfix.deltaFlowCount\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ipfix.flowEndSysUpTime\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ipfix.flowStartSysUpTime\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ipfix.mplsLabelStackLength\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ipfix.octetDeltaCount\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ipfix.packetDeltaCount\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"ipfix.version\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.first_switched\",\"type\":\"date\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.flow_active_timeout\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.flow_end_reason\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.flow_inactive_timeout\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.flow_seq_num\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.flows\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.flowset_id\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.forwarding_status.reason\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.forwarding_status.status\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.icmp_type\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.in_bytes\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.in_pkts\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.last_switched\",\"type\":\"date\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.mpls_label_stack_octets.bottom_of_stack\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.mpls_label_stack_octets.experimental\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.mpls_label_stack_octets.label\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.mpls_label_stack_octets.ttl\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.out_bytes\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.out_pkts\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.sampling_algorithm\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.scope_system\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.total_bytes_exp\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.total_flows_exp\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.total_pkts_exp\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"netflow.version\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"node.hostname\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"node.ipaddr\",\"type\":\"ip\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"sflow.drops\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"sflow.eth_type\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"sflow.frame_length\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"sflow.padded\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"sflow.protocol\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"sflow.sample_pool\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"sflow.sflow_type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"sflow.source_id_index\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"sflow.source_id_type\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"sflow.stripped\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"sflow.sub_agent_id\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"sflow.uptime_in_ms\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"sflow.vlan_priority\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"tags\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true}]","fieldFormatMap":"{\"flow.bytes\":{\"id\":\"bytes\"},\"flow.dst_port\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"http://www.adminsub.net/tcp-udp-port-finder/{{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.dst_port_trans\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"http://www.adminsub.net/tcp-udp-port-finder/{{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.service_port\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"http://www.adminsub.net/tcp-udp-port-finder/{{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.src_port\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"http://www.adminsub.net/tcp-udp-port-finder/{{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.src_port_trans\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"http://www.adminsub.net/tcp-udp-port-finder/{{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.client_asn\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"http://viewdns.info/asnlookup/?asn={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.dst_asn\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"http://viewdns.info/asnlookup/?asn={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.server_asn\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"http://viewdns.info/asnlookup/?asn={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.src_asn\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"http://viewdns.info/asnlookup/?asn={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.vlan\":{\"id\":\"number\",\"params\":{\"pattern\":\"0\"}},\"flow.client_addr\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"https://www.talosintelligence.com/reputation_center/lookup?search={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.dst_addr\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"https://www.talosintelligence.com/reputation_center/lookup?search={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.server_addr\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"https://www.talosintelligence.com/reputation_center/lookup?search={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.src_addr\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"https://www.talosintelligence.com/reputation_center/lookup?search={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.dst_addr_trans\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"https://www.talosintelligence.com/reputation_center/lookup?search={{value}}\",\"labelTemplate\":\"{{value}}\"}},\"flow.src_addr_trans\":{\"id\":\"url\",\"params\":{\"urlTemplate\":\"https://www.talosintelligence.com/reputation_center/lookup?search={{value}}\",\"labelTemplate\":\"{{value}}\"}}}"}}wnoguchi@elastiflow:~/flowtemp$

Download my machine.

https://github.com/robcowart/elastiflow/raw/master/kibana/elastiflow.dashboards.6.3.x.json

wnoguchi@elastiflow:~/flowtemp$ sudo apt policy elasticsearch logstash kibana
elasticsearch:
  Installed: 6.3.0
  Candidate: 6.3.0
  Version table:
 *** 6.3.0 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
        100 /var/lib/dpkg/status
     6.2.4 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.2.3 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.2.2 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.2.1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.2.0 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.1.4 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.1.3 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.1.2 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.1.1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.1.0 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.0.1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.0.0 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
logstash:
  Installed: 1:6.3.0-1
  Candidate: 1:6.3.0-1
  Version table:
 *** 1:6.3.0-1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
        100 /var/lib/dpkg/status
     1:6.2.4-1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     1:6.2.3-1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     1:6.2.2-1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     1:6.2.1-1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     1:6.2.0-1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     1:6.1.4-1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     1:6.1.3-1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     1:6.1.2-1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     1:6.1.1-1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     1:6.1.0-1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     1:6.0.1-1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     1:6.0.0-1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
kibana:
  Installed: 6.3.0
  Candidate: 6.3.0
  Version table:
 *** 6.3.0 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
        100 /var/lib/dpkg/status
     6.2.4 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.2.3 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.2.2 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.2.1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.2.0 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.1.4 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.1.3 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.1.2 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.1.1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.1.0 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.0.1 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages
     6.0.0 500
        500 https://artifacts.elastic.co/packages/6.x/apt stable/main amd64 Packages

Go to Kibana

http://172.16.2.222:5601/app/kibana#/home?_g=()

Navigates

  1. Management
  2. Advanced Settings

[
  {
    "from": "now/d",
    "to": "now/d",
    "display": "Today",
    "section": 0
  },
  {
    "from": "now/w",
    "to": "now/w",
    "display": "This week",
    "section": 0
  },
  {
    "from": "now/M",
    "to": "now/M",
    "display": "This month",
    "section": 0
  },
  {
    "from": "now/d",
    "to": "now",
    "display": "Today so far",
    "section": 0
  },
  {
    "from": "now/w",
    "to": "now",
    "display": "Week to date",
    "section": 0
  },
  {
    "from": "now/M",
    "to": "now",
    "display": "Month to date",
    "section": 0
  },
  {
    "from": "now-15m",
    "to": "now",
    "display": "Last 15 minutes",
    "section": 1
  },
  {
    "from": "now-30m",
    "to": "now",
    "display": "Last 30 minutes",
    "section": 1
  },
  {
    "from": "now-1h",
    "to": "now",
    "display": "Last 1 hour",
    "section": 1
  },
  {
    "from": "now-2h",
    "to": "now",
    "display": "Last 2 hours",
    "section": 1
  },
  {
    "from": "now-4h",
    "to": "now",
    "display": "Last 4 hours",
    "section": 2
  },
  {
    "from": "now-12h",
    "to": "now",
    "display": "Last 12 hours",
    "section": 2
  },
  {
    "from": "now-24h",
    "to": "now",
    "display": "Last 24 hours",
    "section": 2
  },
  {
    "from": "now-48h",
    "to": "now",
    "display": "Last 48 hours",
    "section": 2
  },
  {
    "from": "now-7d",
    "to": "now",
    "display": "Last 7 days",
    "section": 3
  },
  {
    "from": "now-30d",
    "to": "now",
    "display": "Last 30 days",
    "section": 3
  },
  {
    "from": "now-60d",
    "to": "now",
    "display": "Last 60 days",
    "section": 3
  },
  {
    "from": "now-90d",
    "to": "now",
    "display": "Last 90 days",
    "section": 3
  }
]

Navigates

  1. Management
  2. Saved Objects
  3. Import

Import elastiflow.dashboards.6.3.x.json.

Everything done!!

Go to Kibana Dashboard.

Overview

Wao…

Top-N

Very cool…..

Sankey

So Good….

Geo IP

Amazing….

References

tech/network/netflow/elastiflow/elastiflow.txt · Last modified: 2018/07/01 11:50 by wnoguchi