ooni / probe-cli / 7016271537

84%

Build Type

push

github

Committed by web-flow

Commit Message

feat: minipipeline to analyze measurements (#1393)

This diff introduces the `minipipeline` package and command. The general
idea here is to analyze experiments results using an architecture that
matches https://github.com/ooni/data as closely as possible.

The advantage of using this architecture is that we can neatly divide
the data analysis process into three phases:

1. ingestion converts an OONI measurement into observations, which are
flat data structures;

2. analysis converts observations into an analysis structure, which is
significantly simpler than observations;

3. scoring is the process (implemented by each experiment) to turn the
analysis into top-level keys.

That is:

```mermaid
stateDiagram-v2
  state "{ingestion}" as ingestion
  MEASUREMENT --> ingestion
  ingestion --> OBSERVATIONS
  state "{analysis}" as analysis
  OBSERVATIONS --> analysis
  analysis --> ANALYSIS
  state "{scoring}" as scoring
  ANALYSIS --> scoring
  scoring --> TOP_LEVEL_KEYS
```

This diff in particular tackles points 1 and 2. We'll implement 3 for
Web Connectivity LTE in a separate diff.

The diff itself commits several observations and analysis files along
with the measurements from which they originate, which we all
collectively use to write regression tests for the `minipipeline`
package.

The motivation for implementing these changes is that I need a more
clearly defined model to address the differences between Web
Connectivity LTE and v0.4 (see also
https://github.com/ooni/probe-cli/pull/1392). Because there is already
an architectural model for implementing these changes in ooni/data, it
seems logical to borrow liberally from ooni/data.

Incidentally, this change has the benefit of making it easier, in the
future, to align ooni/probe-cli and ooni/data.

The reference issue is https://github.com/ooni/probe/issues/2634.
Implementing this change seems now a precondition for implementing extra
changes that w... (continued)

Run Details

640 of 640 new or added lines in 4 files covered. (100.0%)

4 existing lines in 1 file now uncovered.

24560 of 30576 relevant lines covered (80.32%)

30.34 hits per line