Getting Data Into PNDA

It’s easy to get data into PNDA by design; ingested data should be encapsulated in an Avro schema and published on a pre-defined Kafka topic or set of topics.

avro

Note that this is a big data infrastructure-level data schema, not an application-level data schema; i.e. this is a wrapper which goes around the raw data, hence there is no normalisation fo the data required (more on that in a future post).

There are a number of data sources that we’ve already integrated with PNDA:

Data Type Data Aggregator Data Aggregator Reference PNDA Producer Plugin Reference
BGP (inc. BGP LS) OpenBMP http://www.openbmp.org/#!index.md#Using_Kafka_for_Collector_Integration http://pnda.io/pnda-guide/producer/openbmp.html
BGP PMACCT (BGP listener) http://www.pmacct.net/ http://pnda.io/pnda-guide/producer/pmacct.html
Bulk Ingest PNDA Bulk Ingest Tool http://pnda.io/pnda-guide/bulkingest/
ISIS PMACCT (ISIS listener) http://www.pmacct.net/ http://pnda.io/pnda-guide/producer/pmacct.html
Cisco XR streaming telemetry Pipeline https://github.com/cisco/bigmuddy-network-telemetry-collector
CollectD (CollectD supports multiple plugins as listed here https://collectd.org/wiki/index.php/Table_of_Plugins) Logstash https://www.elastic.co/guide/en/logstash/current/plugins-codecs-collectd.html http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/
IoT sensor via HTTP Node-RED https://nodered.org
Logstash (Logstash supports multiple plugins as listed here https://www.elastic.co/guide/en/logstash/current/input-plugins.html) Logstash http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/
NETCONF Notifications ODL http://www.opendaylight.org/ http://pnda.io/pnda-guide/producer/opendl.html
Netflow / IPFIX Logstash https://www.elastic.co/guide/en/logstash/current/plugins-codecs-netflow.html http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/
Netflow / IPFIX / sFlow pmacct http://www.pmacct.net/ http://pnda.io/pnda-guide/producer/pmacct.html
Openstack Work in progress
sFlow Logstash https://github.com/ashangit/logstash-codec-sflow http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/
SNMP Metrics and Traps ODL https://wiki.opendaylight.org/view/SNMP_Plugin:Getting_Started http://pnda.io/pnda-guide/producer/opendl.html
SNMP Traps Logstash https://www.elastic.co/guide/en/logstash/current/plugins-inputs-snmptrap.html http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/
Syslog Logstash https://www.elastic.co/guide/en/logstash/current/plugins-inputs-syslog.html http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/
Syslog (RFC3164 or RFC5424 – needed for newer IOS/IOS XR/ NX OS etc.) Logstash https://gist.github.com/donaldh/89b7304981f96497c94fe4d98bb03d71 http://pnda.io/pnda-guide/repos/prod-logstash-codec-avro/

If these don’t address your needs, it’s relatively simple to write a custom data producer for PNDA; see the http://pnda.io/pnda-guide/producer/ section in the PNDA guide.  If you’ve written one you want to upstream, we’d welcome your contribution!

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s