Custom Search / Search Builder
The custom search builder module allows you to build a search where the results of the previous stage will feed back into the search for the next stage. Lets have an example.
In the above example, we are performing the following search
- Scrape followers of
- Scrape latest track of each follower
- Scrape all interactions of the latest track of each follower
NOTE: Do be aware that if you are building very complex searches, then this can require lots of requests to the site, which may cause a limiting of the requests you can perform. This likelihood can be increased if you are also performing a filter. There are some ways to try and avoid this, namely setting a random pause after each page of search results are pulled. Also, by providing the specific number of results for each step can help.
Results per Step
From the above search, we are scraping my followers, getting their recent track then scraping the interactions of the track (commenters / likers / reposters). There are a few potential search flows that could be taken depending on the number of total results you may want. If you want 100 accounts returned, if could be you want to scrape 10 followers, then get the recent track from each: 10 tracks then get 10 interactions from each track. Or perhaps you wanted to scrape 100 followers, then the recent track from each: 100 tracks then get 1 interaction from each track. Unfortunately, the program cannot read your mind so it will make a guess at what it was you wanted, unless you tell it.
Double click the specified search step and you can specify how many items you want returned for each. So if you want more results to come from the interactions you can tell it to do so, or if you want to scrape less interactions but from more tracks you can as well.
NOTE: Keep in mind some tracks may not have any interactions, so if you are very specific the program may be unable to fulfill your search, so always leave some leeway for the program to hit your number targets
Export Defined Custom Searches
As always, we try to make things as easy for you as possible, if you have built a custom search and you want to export it for later use, click the EXPORT CUSTOM SEARCH link and it will be saved. When you next open the custom search builder form, you will be able to view your previously saved searches from the dropdown control highlighted above