I had to build infrastructure
for scraping sites which were hostile to it (sorry Cornell ornithology lab, All About Birds, Avibase,
and others). I built infrastructure for coorelating sources, only including a fact like a wingspan
in the deck if it could be corroborated between two sources to within 10%. I built infrastructure
for finding creative-commons licensed images of the birds, AI to find the bird in the image and crop it
well, and build complicated infrastructure for
keeping detailed records of sources and methods, both for debugging, and appropriate attribution.
So far I’ve written 39k lines of code to tackle this problem.