Enterprise Network Observability & Infrastructure Visibility

[+] Status: Completed [+] Origin: Polk State College [+] Date: 2025.08
>> TECH_STACK:
[Netdisco][LibreNMS][Switchmap][SNMP][Ubuntu][PostgreSQL][Perl][Cron][Linux]

Designed and deployed an integrated network monitoring stack to transform a "dark network" into a fully observed infrastructure. Prior to this project, the environment lacked port-level mapping and real-time visibility -relying solely on configuration backup files and physical cable tracing to determine connectivity.

🚫 BEFORE
  • No real-time network visibility
  • Manual cable tracing for port identification
  • Static configuration backups only
  • Unknown device inventory
  • Reactive troubleshooting
→
✅ AFTER
  • Complete L2/L3 topology mapping
  • Instant MAC-to-port queries
  • Real-time performance metrics
  • Automated device discovery
  • Proactive alerting

Three complementary tools deployed on Ubuntu 22.04 VMs provide full-spectrum network observability:

Netdisco
L2/L3 Topology & Device Discovery

Automated device discovery, MAC address tracking, port mapping, and network topology visualization

LibreNMS
Performance Monitoring & Alerting

SNMP polling, bandwidth graphs, uptime tracking, and threshold-based alerts

Switchmap
Directory-Style Port Mapping

Human-readable port status pages, VLAN assignments, and connected device inventory

🔌
Switches Cisco Catalyst (4x enterprise-grade)
📶
Wireless APs Enterprise access points (4x)
🖥
Servers Lenovo SR630s (6x), Dell PowerEdge R720s (~6x)
💻
Virtual Machines ~20 production VMs across hypervisors
🛡
Firewalls Palo Alto Next-Gen (2x)
DATA COLLECTION
📡 SNMP Agents UDP/161
🔍 CDP/LLDP Neighbor Discovery
📋 ARP/MAC Tables L2 Mapping
Poll Data
PROCESSING & STORAGE
Netdisco PostgreSQL Backend
LibreNMS MySQL + RRDtool
Switchmap Perl + HTML Gen
Web Interface
VISUALIZATION
Topology Maps Port Status Bandwidth Graphs Device Inventory

01 Netdisco Configuration

Device Discovery

sudo -iu netdisco ~/bin/netdisco-do discover -d <DEVICE_IP> # Bulk discovery loop for subnet for ip in <IP_RANGE>; do ~/bin/netdisco-do add -d "$ip" -c <COMMUNITY> done

Automated Polling (Crontab)

# Hourly full network refresh 0 * * * * ~/bin/netdisco-do discoverall --force --quiet 10 * * * * ~/bin/netdisco-do macwalk --force --quiet 20 * * * * ~/bin/netdisco-do arpwalk --force --quiet # Daily web frontend restart for stability 0 3 * * * ~/bin/netdisco-web restart >> ~/netdisco_restart.log 2>&1

Schema Migration (v90 → v93)

# Stop services before upgrade ~/bin/netdisco-backend stop ~/bin/netdisco-web stop # Deploy schema updates ~/bin/netdisco-db-deploy # Restart services ~/bin/netdisco-backend start ~/bin/netdisco-web start

02 LibreNMS Integration

Adding Devices via CLI

ssh user@monitoring-server sudo su - librenms ./lnms device:add --v2c -c <COMMUNITY> <DEVICE_IP>

Maintenance Procedures

# Fix daily.sh failures sudo -u librenms -H bash cd /opt/librenms tail -n 200 logs/daily.log ./daily.sh # Clean dirty git tree git reset --hard && git clean -fd ./scripts/github-remove -d git pull --rebase --autostash composer install --no-dev -o

03 Switchmap Deployment

Installation

# Download and extract to web root cd /var/www/html wget https://sourceforge.net/projects/switchmap/files/latest tar zxvf switchmap-*.tar.gz mv switchmap-* switchmap

Perl Dependencies

# Debian/Ubuntu packages sudo apt install libnet-snmp-perl sudo apt install liblog-log4perl-perl sudo apt install liblog-dispatch-perl # Or via CPAN perl -MCPAN -e 'install Net::SNMP' perl -MCPAN -e 'install Log::Log4perl' perl -MCPAN -e 'install Log::Dispatch::Screen'

Configuration (ThisSite.pm)

# Set SNMP community string $Community = '<COMMUNITY_STRING>'; # Or use per-switch community file $CmstrFile = '/path/to/communities.txt'; # Define switches to poll @Ession = ('switch1.domain.local', 'switch2.domain.local');

Cron Jobs (Scheduled Polling)

# Hourly ARP table collection from L3 devices 44 * * * * perl /var/www/html/switchmap/GetArp.pl # Hourly switch scan (MAC tables, port status) 49 * * * * perl /var/www/html/switchmap/ScanSwitch.pl # Daily HTML generation 05 6 * * * perl /var/www/html/switchmap/SwitchMap.pl

Switchmap runs three scripts in sequence: GetArp.pl retrieves ARP tables from routers/L3 switches, ScanSwitch.pl polls MAC address tables and port status via SNMP, and SwitchMap.pl generates static HTML pages served by Apache/Nginx.

04 SNMP Agent Deployment

Standard Configuration (/etc/snmp/snmpd.conf)

# Location and contact metadata sysLocation <DATACENTER_LOCATION> sysContact <ADMIN_NAME> <ADMIN_EMAIL> # Read-only community (no write access) rocommunity <COMMUNITY_STRING> # Listen on management interface agentAddress udp:161

Service Activation

sudo apt update && sudo apt install snmpd snmp libsnmp-dev sudo systemctl enable snmpd sudo systemctl restart snmpd sudo systemctl status snmpd
Operational Efficiency Remote Port Identification
Scenario: Needed to configure switchports for a 5-NIC Proxmox host without physical access
Solution: Used Netdisco's MAC-to-port mapping to query NIC addresses from documentation, instantly identifying which Catalyst ports corresponded to each interface
Impact: Replaced manual cable tracing with a 30-second database query
Systems Administration Database Schema Recovery
Scenario: Netdisco frontend crashes and backend desync after schema drift (v90 to v93)
Solution: Diagnosed version mismatch, stopped services, ran netdisco-db-deploy for schema migration, implemented automated service restarts via cron
Impact: Restored stability and prevented future outages with proactive maintenance
Automation Scheduled Network Polling
Scenario: Manual discovery caused stale data and missed topology changes
Solution: Implemented custom cron jobs for hourly arpwalk, macwalk, and discoverall operations with force flags for complete refreshes
Impact: Network state always reflects reality within 60 minutes of any change
Standardization Unified SNMP Configuration
Scenario: Inconsistent SNMP configurations across physical and virtual assets
Solution: Deployed standardized snmpd.conf across all Linux hosts, configured read-only community strings, and documented the process for future deployments
Impact: 100% SNMP coverage with consistent data quality across all monitored assets
Linux Administration
  • Ubuntu 22.04 server management
  • Service lifecycle (systemctl)
  • User/permission management
Network Engineering
  • L2/L3 topology understanding
  • VLAN configuration
  • MAC address tables
  • Port trunking
SNMP Protocol
  • SNMPv2c configuration
  • Community string security
  • MIB polling
  • Agent deployment
Automation
  • Cron job scheduling
  • Perl CPAN management
  • Bash scripting
  • Service orchestration
Database
  • PostgreSQL administration
  • Schema migrations
  • Query optimization
  • Backup procedures
💻 Virtualization

All monitoring tools deployed on Ubuntu 22.04 LTS VMs running on Hyper-V. Future consideration: migrate to LXC containers for reduced overhead.

🔄 Perl Dependencies

Switchmap requires careful Perl CPAN module management. Used localenv wrapper for Netdisco to isolate Perl dependencies from system packages.

🕐 Polling Intervals

Hourly polling balances freshness with resource usage. Low-change environment doesn't require more aggressive intervals.

🔒 Security Posture

SNMP v2c with read-only community strings. Monitoring VMs isolated on management VLAN with ACL-restricted access.

Monitoring stack deployed and operational
  • ✓ Netdisco topology mapping operational
  • ✓ LibreNMS performance monitoring active
  • ✓ Switchmap port directory generated
  • ✓ SNMP deployed across all physical/virtual assets
  • ✓ Automated polling schedules configured
  • ✓ Database maintenance procedures documented