Simulations are often utilized by researchers to design new algorithms, as testing ideas in the real world can be expensive and risky. However, due to the impossibility of capturing every detail of a complex system in a simulation, researchers typically gather a small amount of real data that they can replay while simulating the components they want to study.
To improve the accuracy of these simulations, a team of electrical engineering and computer science graduate students and senior authors at MIT have developed a new machine-learning algorithm that incorporates the principles of causality to learn how the data traces were affected by the behavior of the system. This approach enables researchers to replay the correct, unbiased version of the trace during the simulation.
When compared to existing simulators that do not account for bias, the researchers’ simulation method accurately predicted the best algorithm for video streaming in terms of less rebuffering and higher visual quality. The study’s co-lead author, Arash Nasr-Esfahany, emphasized the importance of understanding the underlying data generation story when answering counterfactual questions, stating that “data are not the only thing that matter.”
The research was presented at the USENIX Symposium on Networked Systems Design and Implementation and was conducted by Abdullah Alomar, Pouya Hamadanian, Anish Agarwal, Mohammad Alizadeh, and Devavrat Shah.
Specious simulations
MIT researchers have conducted a study on trace-driven simulation in the context of video streaming applications. In video streaming, adaptive bitrate algorithms decide the video quality to transfer to a device based on real-time data on the user’s bandwidth. To test the impact of different algorithms on network performance, researchers collect real data from users during a video stream for a trace-driven simulation.
However, researchers have traditionally assumed that trace data are exogenous and not affected by factors changed during the simulation, resulting in biases about the behavior of new algorithms. To address this issue, the researchers framed it as a causal inference problem and developed a solution that involves disentangling the intrinsic elements that affect network performance from the effects of the actions taken by the algorithm.
By understanding what aspects of the behavior are intrinsic to the system, the researchers can perform unbiased simulations. This approach improves the accuracy of simulations, enabling researchers to make better decisions when designing and testing algorithms for video streaming applications.
Learning from data
MIT researchers have developed a new tool called CausalSim, which solves the problem of biased trace-driven simulation. Typically, trace data collected during a randomized control trial are assumed to be exogenous, but this assumption is often false, leading to errors in simulation results. CausalSim solves this problem by disentangling the intrinsic properties of a system from the effects of actions taken, allowing for unbiased simulations.
CausalSim uses trace data to estimate the underlying functions that produced those data. This allows researchers to determine how a new algorithm would change the outcome under the same underlying conditions that a user experienced. The algorithm consistently improved simulation accuracy, resulting in algorithms that made about half as many errors as those designed using baseline methods.
In a video streaming example, CausalSim helped researchers select an improved bitrate adaptation algorithm with a lower stall rate, while an expert-designed trace-driven simulator predicted the opposite. The researchers confirmed the accuracy of CausalSim by testing the algorithm on real-world video streaming.
In the future, the researchers plan to apply CausalSim to situations where randomized control trial data are not available and explore how to make systems more amenable to causal analysis.