Telling a chain's national menu from one pub's own
How WhoozGot recognises a chain's national menu page shared across branches and keeps it apart from each pub's own price list.
What I tried
A national chain publishes one menu page for all its branches. A crawler meets it once per branch. About a quarter of all documents turned out to be a brand's national page rather than one venue's list.
Why I chose differently
A brand's page looks unique the first time it is met. It only reveals itself as shared once more branches turn up, so it cannot be decided while crawling.
What I chose
A sweep over the whole dataset now sets shared pages apart. They are kept, because a chain's national offer is still useful, but never treated as one pub's prices. Small independent groups that genuinely share a menu are respected. Anything a person has already ruled on is left alone.
The principle
Some facts about a record only exist across the whole dataset. Decide them in a pass over everything, not while collecting.
- Web crawling
- Deduplication
- Data quality
- Pub data