How To: Enriching IPs With Python

I need a certain amount of enrichment on IP addresses multiple times per day. While certain tools and controls have built-in enrichment, I often find myself sitting at a shell prompt needing the same or similar info quickly.

So, I decided to write something that will take an IPv4 address as an argument, and return the data I need in an easily-parsable format.

The code is available here.

Usage

The code takes an IPv4 address as input, and outputs the following:

  • IP: IP address, taken from argv[1]
  • GeoCC: country code according to MaxMind GeoIP2 country database
  • ShoCC: country code according to Shodan
  • rvAS: ASN, per RouteViews MRT/RIB BRP database
  • ShoAS: ASN from Shodan
  • rvCIDR: CIDR range for the AS, per RouteViews
  • ShoORG: AS Organization name, per Shodan
  • DNS: Reverse DNS lookup result
  • ShoDNS: hostnames for IP, per Shodan
  • ShoPorts: Open ports for IP, from Shodan
  • ShoTags: Any associated Shodan tags
  • ShoVulns: Any known positive or negative vulnerability test results, per Shodan

Requirements

How To

You’ll need a copy of the RouteViews database. Thankfully, Pyasn comes with utilities that make this a snap.

The following will download the latest copy:

$ pyasn_util_download.py --latest
Finding most recent archive in /bgpdata/2017.12/RIBS ...
Downloading ftp://archive.routeviews.org//bgpdata/2017.12/RIBS/rib.20171201.1600.bz2
 100%, 58KB/s
Download complete.

Now, it must be converted into a usable format:

$ pyasn_util_convert.py --single rib.20171201.1600.bz2 ipasn.dat
Parsing MRT/RIB archive ..  MrtTD2Record (PEER-INDEX-TABLE, collector 2162111334, 62 peers)
  MRT record 100000 @14s
  WARNING: repeated prefix '65.254.166.0/24' maps to different origin (62793 vs 22299)
  WARNING: repeated prefix '65.254.181.0/24' maps to different origin (62793 vs 22299)
  WARNING: repeated prefix '65.254.182.0/24' maps to different origin (62793 vs 22299)
  WARNING: repeated prefix '65.254.183.0/24' maps to different origin (62793 vs 22299)
  MRT record 200000 @28s
  WARNING: repeated prefix '103.80.80.0/22' maps to different origin (56258 vs 58463)
  WARNING: repeated prefix '103.80.80.0/23' maps to different origin (56258 vs 58463)
  WARNING: repeated prefix '103.80.82.0/23' maps to different origin (56258 vs 58463)
  WARNING: repeated prefix '103.251.199.0/24' maps to different origin (45561 vs 198949)
  MRT record 300000 @42s
  WARNING: repeated prefix '159.230.0.0/16' maps to different origin (62793 vs 22299)
  WARNING: repeated prefix '164.102.0.0/16' maps to different origin (36534 vs 16623)
  MRT record 400000 @57s
  WARNING: repeated prefix '177.55.40.0/22' maps to different origin (53115 vs 262482)
  MRT record 500000 @74s
  WARNING: repeated prefix '198.161.9.0/24' maps to different origin (36534 vs 16623)
  WARNING: repeated prefix '198.161.10.0/24' maps to different origin (36534 vs 16623)
  WARNING: repeated prefix '198.161.11.0/24' maps to different origin (36534 vs 16623)
  WARNING: repeated prefix '198.161.230.0/24' maps to different origin (36534 vs 16623)
  WARNING: repeated prefix '198.161.231.0/24' maps to different origin (36534 vs 16623)
  MRT record 600000 @90s
  WARNING: repeated prefix '203.32.185.0/24' maps to different origin (55871 vs 17755)
  MRT record 700000 @104s
IPASN database saved (731953 IPV4 + 0 IPV6 prefixes)

Place the resulting ipasn.dat file wherever you’d like to store it, and ensure that the PATH variable in the ipinfo.py code reflects this location.

Next, you’ll need the MaxMind GeoLite GeoIP2 Country database.

$ wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz
--2017-12-01 14:36:40--  http://geolite.maxmind.com/download/geoip/database/GeoLite2-Country.tar.gz
Connecting to 127.0.0.1:9050... connected.
Proxy request sent, awaiting response... 200 OK
Length: 1611049 (1.5M) [application/gzip]
Saving to: ‘GeoLite2-Country.tar.gz’

GeoLite2-Country.tar.gz               100%[===============================================================================>]   1.54M  2.44MB/s    in 0.6s    

2017-12-01 14:36:41 (2.44 MB/s) - ‘GeoLite2-Country.tar.gz’ saved [1611049/1611049]

Next, uncompress the archive and move the GeoLite2-Country.mmdb file into the same location you stored the ipasn.dat file above.

$ tar xf GeoLite2-Country.tar.gz 
$ cd GeoLite2-Country_20171107/
$ ls
COPYRIGHT.txt  GeoLite2-Country.mmdb  LICENSE.txt

Now edit ipinfo.py, and add you Shodan API key. You’ll need a valid Shodan account. You’ll find it in your account settings.

Once you’ve done this, you should be able to get quick info on any IP:

$ ipinfo 8.8.8.8
IP GeoCC ShoCC rvAS ShoAS rvCIDR ShoORG DNS ShoDNS ShoPorts ShoTags ShoVulns
8.8.8.8|US|US|15169|AS15169|8.8.8.0/24|Google|google-public-dns-a.google.com.|[u'google-public-dns-a.google.com']|[53]|[]|None

Rationale

I’d originally intended to show two variants of this script: One using the locally-stored databases, and one using Shodan. However, you should never trust a single source for your data. Particularly when it comes to things like geolocation. So, I decided to combine the two approaches into a single script.

Personally, I comment out the line that prints the data header, so I can call it from other scripts when processing a large number of IPs at once, to get a more easily-parsable output.

Caveat

Be aware that this may use your various Shodan tokens. So keep an eye on them while testing, to ensure you’re aware of how the script might affect your Shodan tokens.

HowToBlue Team