Moving intelligence closer to a machine changes the design problem. Compute, memory, energy, connectivity and response time become part of the same discussion as model capability.
It is tempting to describe the task as making a model smaller. Compression may be an important technique, but size alone does not explain whether the resulting system is useful.
Start with the operation
The first useful question is what the system needs to do. A machine performing a bounded task may require a particular part of a model’s broader capability. It may also need a predictable route from an observation to a response, with explicit handling of uncertainty and failure.
That operational context determines the trade-offs worth investigating. A system with intermittent connectivity has a different set of design constraints from one with a reliable network connection. A device with a fixed energy budget faces different choices from a machine connected to continuous power.
These are properties of the intended deployment, rather than details to be added after a model has been selected.
Measure beyond the model
An evaluation of edge intelligence should connect model behaviour to the device and workload. Depending on the task, relevant observations might include:
- End-to-end time between receiving an input and completing the required response.
- Memory use during the actual execution path.
- Energy consumed across a representative operation.
- Behaviour under changing inputs, incomplete observations and interrupted services.
- The ability to identify and recover from an incomplete operation.
A reported inference time can be useful. It is still different from the time taken by a complete system to sense, decide, act and establish what happened.
Improvements can move a bottleneck
A change that reduces one cost may increase another. A compressed model may need more processing around its inputs. A more capable device may change energy or thermal constraints. A local execution path may reduce network dependence while increasing the importance of device management and versioning.
The right unit of evaluation is the connected implementation. Improvements should be assessed against the task and its operating conditions, with the relevant trade-offs made visible.
Our interest
Conexiq’s developing research programme treats edge intelligence as an intersection of models, software and physical platforms. We want to investigate how capability can be adapted to a task and evaluated on the hardware where it is intended to operate.
This perspective keeps the purpose of the work clear: a smaller footprint is valuable when it helps the complete system perform a useful operation within its real constraints.